DEV Community: Tracy Gilmore

Using a WebWorker in the browser to create a plug-in architecture

Tracy Gilmore — Thu, 30 Oct 2025 21:07:27 +0000

Very few business information systems these days use anything other than the web platform as the user interface. It is also regarded as wise to only include presentation logic in the frontend; pushing as much business logic into the backend as possible. However, there are always exceptions and in some specialist applications such "rules" are occasionally bent for sound architectural reasons.

An increasingly popular architecture these days is the JAMStack where the frontend is hosted in the same way as a static website (via a CDN for example) but with more dynamic behaviour. In this architecture there tends to be some specific business logic in the frontend with the "heavy lifting", such as storage and authentication, provided in a generic manner through third-party APIs.

This novel application design offers a great deal of resilience and flexibility but not necessarily the extensibility that can be achieved using a plug-in architecture. When an application requires greater extensibility or the ability to call on functionality not necessarily developed as part of the original application, or even by the same developers, a plug-in architecture is a candidate solution. As described below, there are several possible approaches but using a web worker can provide significant benefits.

In order to discuss a variety of approaches around a common context we will start by describing an example use case (application) used to exercise and demonstrate.

Test scenario

In this very simple example (honestly I do not think I could envisage a simpler) we will perform the four basic mathematical operations (addition, subtraction, multiplication, division) on two numbers and return the answer. Plugins will provide the actual arithmetic operations.

The 'application' will provide the inputs and present the output. In addition the program will;

load the list of available plugins (manifest)
present buttons for each plugin
when the button is pressed, the plugin will be loaded, passed the inputs and execute with the output returned for presentation.

All four plugins will expose a single 'calculate' function that accepts two numeric inputs and returns the result.

Example Plugin: addition.js

function calculate(num1, num2) {
  return num1 + num2;
}

List of plugins (manifest.json)

[
  {
    "pluginName": "Addition",
    "pluginFile": "addition.js"
  }

]

A working example can be found in my web-worker GitHub repository.

Exploring candidate solutions

How can a plug-in architecture be achieved when desktop applications employ web technologies and, more specifically, operate in the web browser. The answer, or at least part of it, is the Web Worker API. Granted, I am discussing a rather exotic requirement in the wide world of web applications but as the capability of the web browser increases such potential solutions become possible.

In a plugin architecture it is usual for the application to load a specific file at runtime, which we will call the manifest, that details what plugins are available and how to find/load them. This file could be sourced from outside the application easily using a AJAX/XHR request and having the data in JSON format.

Before making the plug-in model available for 'foreign' developers the application developers need to establish a Software Development Kit (SDK). As a minimum the SDK should be a document outlining the API ('api'), which outlines the interface the application expects all plugins to support. In addition, the SDK can provide test code developers can use to exercise and validate their plugin before using it with the application.

Some less effective alternatives

Before we dive deeper into the technicalities of the Web Worker mechanism, we will explore a couple of ways in which similar functionality can be achieved and their limitation.

At the crux of the following examples is need for a mechanism to load scripts on demand and with the name of the file/module supplied at runtime. The name of the "plugins" are not know at build time, but supplied through a file loaded at runtime, and certainly not included in the application bundle.

In the example code we have to demonstrate each of the approaches described below there is a single application (index.html) with a script element for each approach. For each script there is a loosely-coupled plug-in (JS) file.

index.html

<body>
    <script src="dom-injection.js"></script>

    <script type="module" src="dynamic-import.js"></script>
</body>

DOM inject

dom-injection.js

console.log('dom-injection');

function loadScript(scriptPath, functionName) {
    window[functionName] = () =>
    console.log(`${functionName} is not yet loaded`);

    const scriptElement = document.createElement('script');
    scriptElement.type = 'text/javascript';
    scriptElement.async = true;
    scriptElement.src = scriptPath;
    scriptElement.onload = () => {
        console.log(`${functionName}() is now ready`);
    };
    document.body.appendChild(scriptElement);
}

loadScript('./hello-world.js', 'helloWorld');

hello-world.js

function helloWorld() {
    alert('Hello, World! (injection)');
}

1) injection of script tags into the DOM
2) Pollution of the global namespace

Dynamic imports

dynamic-import.js

console.log('dynamic-import');

async function loadScript(path, functionName) {
    window[functionName] = () =>
        console.log(`${functionName} is not yet loaded`);

    const module = await import(path);

    window[functionName] = module.default;
    console.log(`${functionName}() is now ready`);
}

loadScript('./hello-world.module.js', 'modHelloWorld');

hello-world.module.js

function helloWorld() {
    alert('Hello, World! (import)');
}

export default helloWorld;

1) Web bundlers
2) Short lived / not cacheable

Both: execution in the primary thread

Output

Viewing the index.html in the web browser will produce the following output in the developer console.

This shows the two scripts loading followed by the two loadScript functions running to load the stipulated plugin script and execute the named function it contains. In both scenarios the function is initialised with a temporary version that is never executed but prevents an error being reported should the function be called before it is loaded.

A worked example (Web Worker)

Let's start by listing the 'moving' parts:

The application that will utilise the plugins
The manifest file that details what plugins are available at runtime
The web-worker code that will load, execute (and potentially cache) plugins
Finally, the plugins themselves.

The Scenario

We will use simple mathematical operations, each their own plug-in, to perform Celsius and Fahrenheit temperature conversion. To recap, here are the calculations.

Fahrenheit = (Celsius * 9) / 5 + 32

Celsius = (Fahrenheit - 32) * 5 / 9

These are the test cases we will use to exercise the process.

-40°C => -40°F
0°C => 32°F
32°F => 0°C
50°C => 122°F
122°F => 50°C
100°C => 212°F
212°F => 100°C

The main application file will load the test cases as a JSON file and initialise the primary Web Worker. Each test case will be sent one-by-one to the primary worker for calculation and the response output by the application code. The initial call to the primary worker will include a reference to the manifest that will be loaded in preparation for subsequent calls. For each test case called three plug-ins will be used, with the first use also involving loading of the plug-in itself.

Exercising each test case will involve splitting the input string into its measurement and scale components. The measurement will be converted into a number whilst the scale will identify which formula is to be adopted and therefore which plug-ins will be called.

NB: In the following code example I have deliberately excluded any defensive code to keep it on topic. The source code for this project can be found in this GitHub repo and comprises of around a dozen files.

Simplified solution

Prior to demonstrating the web worker-based plug-in solution we will review the script section of the all-in-one sampler.

const testData = [
    ['-40°C', '-40°F'],
    ['0°C', '32°F'],
    ['50°C', '122°F'],
    ['100°C', '212°F'],
];

const add = (x, y) => x + y;
const sub = (x, y) => x - y;
const mul = (x, y) => x * y;
const div = (x, y) => x / y;

console.table(
    testData.flat().map((testDatum) => ({
        testcase: testDatum,
        result: convertTemperature(testDatum),
    }))
);

function convertTemperature(datum) {
    const cToF = (t) => add(div(mul(t, 9), 5), 32);
    const fToC = (t) => div(mul(sub(t, 32), 5), 9);
    const [temp, scale] = datum.split(/°/);
    return scale === 'C' ? `${cToF(+temp)}°F` : `${fToC(+temp)}°C`;
}

In the above source code we have the test data as an array at the top, followed by the four arithmetic operations we will contain as plug-ins.

The next section exercises the convertTemperature function using the test cases and presents the results as a table in the browser console.

Finally, we have the convertTemperature function itself that defines the two conversion calculations, using the arithmetic operations. The function splits the input test case at the degree symbol ° into the temperature unit and scale sections. The temperature unit is then cast from a string into a number and the scale is used to select the calculation required.

In the demonstration code we extract the test data from the sampler into its own JSON file, each arithmetic operation into its own script file, to be loaded by the web worker, the convertTemperature function into the primary web worker plug-in.js file.

Manifest

One significant variation from the scenario described initially in this article and the demonstration code, is the structure of the manifest, which has been simplified.

{
  "add": "addition.js",
  "sub": "subtraction.js",
  "mul": "multiplication.js",
  "div": "division.js"
}

A more significant difference between the two examples is that the simple sampler processes the test cases sequentially but the demonstration application processes them asynchronously.

Application (Index.html script)

The script within the application index.html file starts with loading the testData and preparing the results array. The plug-ins.js file is then loads the web Worker and issues a postMessage to initiate it with a reference to the manifest file. The bulk of the code is contained within the onmessage listener of the web worker.

plugInWorker.onmessage = ({data}) => {
  if (data === "__LOADED__") {
    testData.flat().map(
      (testDatum) => plugInWorker.postMessage(testDatum)
    );
  }
  else {
    results.push(data);
    if (results.length === testData.flat().length) {
      console.table(testData.flat().map(
        (testDatum, index) => ({
          testcase: testDatum,
          result: results[index]
        })
      ));
      plugInWorker.terminate();
    }
  }
};

The onmessage listener anticipates receiving two types of message:

__LOADED__ indicates the manifest has been processed by the plug-ins worker and all the operation scripts have been 'loaded'.
result is the output from each calculation. When all the calculations have been received they are presented via the browser console.

Plug-ins.js

In addition to the convertTemperature function, the plug-ins.js file contains the instructions to load the manifest file and the scripts containing the arithmetic operations.

let manifestLoaded = false;

onmessage = ({ data }) => {
  if (manifestLoaded) {
    postMessage(convertTemperature(data));
  } else {
    manifestLoaded = true;
    import(data, { with: { type: 'json' } }).then(
      ({ default: manifest }) => {
        importScripts(
          ...Object.values(manifest).map(
            (file) => `./plug-ins/${file}`
          )
        );
        postMessage('__LOADED__');
      }
    );
  }
};

The key aspects include the following:

The onmessage listener is configured to receive requests from the application.
The postMessage instruction is used to send messages back to the application.
A dynamic import is used to load the operation scripts defined by the manifest data.
The scripts referenced in the manifest are loaded using the importScripts instruction.

Word of Caution

This kind of architecture should only be considered for use in a controlled environment and only where the authors of the plugins can be trusted. Failing to take adequate precautions would have the potential of exposing the user to considerable risk.

An NPM dependency check list

Tracy Gilmore — Sun, 21 Sep 2025 14:08:17 +0000

Bringing in a third-party package to your project might save you time and effort but could be a mixed blessing, as demonstrated by recent supply-chain attacks perpetrated via packages in the NPM registry.

The potential risk posed by the unknown nature of such packages reminds me of the Schrodinger's Cat thought problem but not because of the cat being both alive and dead. Instead of asking what state the cat will be when I open the box, I wonder with it be a pussy-cat or a ferocious tiger, poised to pounce and bit my head off? A bit extreme I agree but my point is that adding code to your project without checking it thoroughly is a risky strategy.

Every time I hear the phrase "but why 'reinvest the wheel'?" it always strucks me a odd given just how many varieties of wheel there are (see hero graphic). I don't know about you, but I know which variant I would prefer my aeroplane to use!

Modern software development is heavily dependent on packages, whatever the technology stack. What proportion of your project is bespoke code and not there just to glue packages together, not much I would bet. All languages that employ 3rd-party packages (NuGet, Mavern, Pypi, crates.io, Composer) are vulnerable but NPM (and JSR), and the JS ecosystem, is particularly exposed.

This is why I drafted the following check list. Before increasing our exposure I have decided to ask some questions about candidate packages prior to inclusion.

My NPM dependency checklist

1 Read the ReadMe and the associated website (if one is provided) to confirm it meets your specific use cases, and is not more than you need.

2 Does the required version contain any known vulnerabilities? A useful tool for checking this is the Snyk Vulnerability Database.

3 Does the package employ an acceptable (Open-Source) licence. First look at NPM and if unsure, check the OPS.

Another place to check in the Open Source Initiative.

4 Other things to check on the NPM page (section illustrated above) include:
• Is it a populate package (weekly downloads)?
• Is it a major version?
• Is the size of the package acceptable?
• Is it being actively maintained?
• Is it being maintained by a single developer or a group of collaborators?

5 Check dependencies and version history.

• Is the very latest version possibly a little too cutting-edge, is there a slightly earlier version that is still being maintained?

• When was the last update?
• Are there too many or any concerning dependencies?

6 If you have more than one option, check how frequently each are downloaded using something like npm trends.

7 If hosted in GitHub, check how many issues there are and how frequently issues are being resolved.

8 Also, check how active the discussion section is and if there is a security section, check that out as well.

9 Finally, before committing to integrating the package, if there are any doubts it might be worth checking the package with the OWASP Dependency Checker

Road to self-driving development

Tracy Gilmore — Wed, 20 Aug 2025 18:11:18 +0000

There have been years of investigation into automated systems for everything from aircraft, weapon systems and unmanned drones to automobiles. Despite the millions, if not billions, of dollars invested and claims made by some electric vehicle manufactures, the day of fully-automated (aka self-driving) cars is a long way off.

I would argue that handing over the responsibility for developing the systems on which our society (our civilisation) is so reliant, is an exceptionally dangerous strategy. Far more dangerous than streets full of driver-less vehicles, yet we seem to be determined to hand over this vital responsibility, according to some big businesses.

Call them Software Developers or Software Engineers, it makes little difference if the AI hype is to be believed and all our jobs are at risk in the next 2-5 years. I see some dangerous parallels between the rush to self-driving cars and the use of AI in the Software Development Life-Cycle.

Six levels of Automation

Several studies into automation have defined a 6-level scale of automation that I think might be applicable to the software industry’s adoption of AI and the risk it might present.

In the context of software development the 6 levels can be aligned as follows:

Level 0: The developer codes everything using a text editor but not an Integrated Development Environment (IDE.)
Level 1: The IDE provides developers with functions such as auto complete and/or syntax highlighting.
Level 2: The IDE integrates features like linting and always running test execution.

Up to this level the developer is completely in control and decides how to use the information presented by the assistive features. They are responsible for determining what course of action should be taken.

Level 3: Current AI tools can perform analysis of the source code and can suggest improvements the developer can choose to accept or reject.
Level 4: When prompted by the developer, the AI tools can go beyond the current code base and draw on information from external resources, within the system domain (MCP) and outside (training material), to generate new code.

The next level is, in my uninformed opinion, the scary one, that potentially puts our society at risk. Cutting expensive human resources is an essential strategy in many businesses but, in this context, is the only cost the jobs of software developers?

Level 5: The responsibility for development of the system is completely controlled by the AI tools without any developer intervention. At this point we have no idea what the system is doing and cannot be absolutely sure AI has our best interest in “mind”, or whatever the AI equivalent of mind might be.

Should we really hand over this responsibility without question just because big business says “trust us”?

Resources

Aviation Autonomy, Summary of the Consultation Paper, UK Law Commission: JARUS levels of automation, page 6.
Synopsys, The 6 Levels of Vehicle Autonomy Explained
NHTSA, Automated Vehicles for Safety
Navigating the 5 Levels of Drone Autonomy: A Look at SUIND’s Approach to Autonomous Systems
DroneLife: DRONEII: Tech Talk – Unraveling 5 Levels of Drone Autonomy

Experiments with
and
elements

Tracy Gilmore — Sat, 02 Aug 2025 19:21:23 +0000

The details and summary elements are relative newcomers to the HTML canon and I was curious what mischief I could get up to with them. The spoiler is quite a lot but their is an aesthetic limitation to be overcome before any of my experiments could become practicable.

To start, let's take a quick tour of the HTML and some of its hidden features. On the face of it there is a element that wraps both a element, that is always shown, along with whatever you want present when the details element is open.

    <details>
      <summary>Summary</summary>
      <div>Content</div>
    </details>

Although this can be multiple elements I have found containing everything other than the summary element makes things easier to style.

The default behaviour of the details element is to toggle between hiding and showing the inner content when the summary is clicked, without the need for any JS code. In the later experiments there is some JS code to achieve more functionality but a great deal can be done without.

Codepen basic details/summary example.

There are two pseudo elements employed by the details and summary elements. The most obvious one is ::marker which is the triangle usually shown before any text (or other content) of the summary element in the top left-hand corner. Toggling the content will also cause the marker to rotate back and forth, again no JS required.

The second, and actually more useful pseudo element is ::details-content and can be used to reference immediate children of the details element other than summary. This comes in very handy for styling the content when in the open state, i.e. when the details element has an open attribute assigned.

In the first proper (term used advisedly) experiment we will also take advantage of more built-in behaviour of the details element. The name property, when the same value is applied to more than one details element, ensures (without JS) only one ::details-content is presented at a time. Thus, we can easily create a form of accordion component only using HTML and CSS.

Codepen Accordion example.

At this point the ::details-content has always been in close proximity of the summary element, in fact immediately following (below) but it does not have to be. By applying a position other than the default static we can configure other regular components. Clicking on the summary toggles the open state of its details parent but clicking the ::details-content requires a little JS code to have the same effect. We could wrap the content inside the summary and apply some addition CSS styling to save us using JS but we would be unable to use the ::details-content pseudo elements, it is quite a distortion of the components purpose and is bound to have an adverse impact on the accessibility of the mark-up.

Codepen Tabs example.

In the tabs example we align all the summary elements horizontally at the top of the component. The content panel (::details-content) is positioned immediately below thesummary`s across the full width of the component. All this can be achieved using CSS alone.

Codepen Ribbon example.

The ribbon is similar to the tabs component but with a short panel containing selectable options. There is a little JS code in this example that is required to close all the details elements when an option in the content panel is selected (clicked or keypress).

In the final example we use the composition of several details to construct a multi-level menu.

Codepen menu example.

The aesthetic limitation

The expansion/collapsing behaviour of the details/summary is quite abrupt, snapping open and closed. At this point in time it does not appear to be possible to animate the behaviour for a smooth presentation.

Just an idea: Using a prefix with id attributes for direct JS access

Tracy Gilmore — Thu, 17 Jul 2025 11:43:32 +0000

A little-known fact is that valid DOM element id attributes are exposed by the DOM into the JavaScript space as properties of the window (globalThis) object. This means an element such as:

<div id=”testDiv”>Hello, World!</div>

Can be accessed as follows without the need to use document.querySelector(‘#testDiv’) or its older cousin document.getElementById(‘testDiv’).

console.log(testDiv.textContent); // “Hello, World!”

However, as is, this pollutes the global namespace and MDN cautions against using the id attribute this way primarily because of the potential conflict with other entities in the global namespace such as window properties. MDN advises using the aforementioned document methods.

When hand-crafting CSS, a long-lost art, the general (and sound) advice is to avoid using the id of an element to apply styling as this could be too specific. Personally, I have not had too much of an issue with styling by id but usually only in hobby projects and prototypes. In production code I usually stick with the guidance and would recommend the guidance to fellow frontend developers.

But, what if we could force the CSS issue whilst considerably reducing the potential of global namespace collision? Whilst testDiv is a valid CSS selection by id #testDiv, adding a dollar symbol to the front makes it invalid (#$testDiv is not a valid CSS selector).

At the same time, it is highly unlikely that the window API will acquire properties with such a prefix, reducing collision potential.

There are JS libraries (of old) that use the $ symbol, the most well-known of course being JQuery, and whilst there is a wealth of code out there in the wide using such libraries, I suspect they are not used anywhere near as much in production code these days.

In conclusion, prefixing element id attributes with a dollar symbol would:

Impede the use of the id in CSS styling, not that it is done very often these days.
Enable JS to reference the DOM element directly without the need to look up the element first using a document method such as querySelector or getElementById.

<div id=”$testDiv”>Hello, World!</div>

console.log($testDiv.textContent); // “Hello, World!”

Is this a crazy idea? I would like to hear what others have to say in the comments below.

I’ve given up on Windows and you could too

Tracy Gilmore — Sat, 25 Jan 2025 10:39:44 +0000

The recent instability caused by Windows updates and the shenanigans over Recall has pushed me to switch to Linux, and I will not be looking back any time soon

I will start this post with a few words of caution. If you are happy with Windows on your personal PC and the potential risks posed by the (forced) introduction of the Recall feature, stick with Microsoft. Switching to Linux is not something that should be done without due consideration, but it is now easier than I think it has ever been and there are more reasons than ever to consider the jump.

Back story

I am an experienced (old) software engineer and have long has a laptop PC of my own to learn new technologies and develop personal projects. My employer (a software firm) has preferred to only use MS Windows on PCs, largely because of the administrative control and perceived control it offers. Linux was only really considered for client servers where the cost of MS Windows Server becomes a compelling negative factor.

I have always liked Linux and have investigated it several times over the years but the following factors have usually tilted the balance in Microsoft's favour.

New laptops tended to come with the latest edition of MS Windows installed, so the licence cost was wrapped up in the price of the PC, whether you like it or not.
Updates used to be reliable and even recent upgrades (Win-7 to 8 to 10 to 11) were free as long as your machine was compatible.
It was preferable to maintain a degree of compatibility with what I used at work but that was more an issue in the day of desktops and working in an office. I have not done either for more than five years.
Many of the office tools I used to use were Windows only but since the move of such tools to the web this is no longer a constraint. Also, my need to use exactly the same tools to access documents on my personal PC has vanished.

So why (and how) did I change to Linux

The primary reason for abandoning MS Windows was the instability caused by recent updates and how all my attempts to wind back the update failed. This opened the door ajar for the change. I watched a few videos to gauge which variant I should choose and decided on Linux Mint Cinnamon 22.

So, I downloaded a copy and prepared a USB key to trial it. I was very impressed and it lived up to all the commentators had said but I was not quite ready to make the leap.

The secondary push

My wife had an old laptop that was no longer compatible with Windows but wanted to start using again.

I brought out the USB key I had prepared earlier (Blue Peter reference for older British readers) and within minutes we had installed Linux on the primary (SSD) drive, leaving all her data files safe on a secondary (HDD) drive.

I can honestly say the process was quick and painless, even with the minor configuration changes for use by a non-US user.

Then came the fringe (but not insignificant) benefits. Not only was the Operating System (OS) incredibly easy to use and fully featured, it was very responsive and the battery usage was much better.

I was convinced it was time to change

Just when I thought I had resolved the Windows update problem I was presented with another Blue Screen Of Death. I was sure there was no hardware fault and the cause was all down to the misbehaving OS. I'm done. Windows is history. Time for me to “bite the bullet” as they say in The States and drop Windows for Linux.

I was “pushing on an open door”, but the experience of building (that makes it sound more difficult than it was) my wife's laptop demonstrated how simple the process would be.

Restart the PC to enter the BIOS settings.
Ensure the USB key was the primary boot source.
Push the USB key in and reboot.
Follow the step by step guide and you are done.
When prompted, remove the USB key and reboot into Linux.

Closing thoughts

We have now been using the new OS for a month and have absolutely no regrets. In fact there have been some pleasant surprises.

The entire OS installed within 16MB on my primary drive, a lot less than Windows.
It is fast and responsive and I have yet to see a greyed out waiting screen because the OS is busy.
Battery life - wow. OK, a few months before all this I had bought a new battery and had been using it with Windows so was used to the performance boost it provided (a couple of days from hours). But I was not expecting Linux to be nearly twice as efficient with the battery power (nearly a week between charges.)

There has been some adjustment needed, primarily in using the command line. I am more familiar with shell script than my wife but she had become frustrated with Windows’ CMD v PowerShell. Whilst she now has to learn a new scripting language, there is only one and the need to use it is not frequent.

I keep asking myself “why did I not do this earlier” and it comes down to the combination of push and pull.

Push: MS Windows is too problematic.

Pull: Linux Mint Cinnamon is soo easy.

If you liked this post, you might find my post on using a chromebook for development of interest.

Using a tagged template to simplify Regular Expressions

Tracy Gilmore — Thu, 05 Oct 2023 17:15:32 +0000

There is a joke that goes:
Alice: "Did you resolve that string array filter problem?"
Bob: "Yes, I used a Regular Expression."
Alice: "Great! Now we have another problem!"
If you know RegExp, you also know the truth of Alice's words.

Introduction

Regular Expressions have a justifiable reputation for being complicated and indecipherable. So why are they present in virtually every imperative programming language? Probably because they are so powerful and efficient at parsing text, which is particularly useful for finding and replacing matching text. But still, its terse encoding can be difficult to comprehend beyond a dozen characters used for something like splitting a string into an array. It can take a surprisingly short period of time for even the original author of a pattern, before they find it difficult to interpret the magical symbols.

Maintaining and testing RegExp can also be challenging because of all the test case permutations some RegExp patterns require. It is about as far from self-documenting code as, I think, a high-level language can get. So how can we improve the readability, and therefore the testability and maintainability, of RegExp patterns?

Template Literals to the rescue

If you have not switched from using regular Strings to Template Literals (TL) in your code, you are probably missing a trick. There can do everything regular strings can do but also have the following “super-powers”:

No more flame wars over the use of single or double quote delimiters. Besides the fact the team should have agreed on a convention that is applied through tooling (such as prettier), TLs have only one option - backticks (ASCII character x60, aka grave accent). Out of the three delimiter options it now seems rather odd to choose between two that have alternative uses (apostrophes and inch symbol) when the backtick is seldom used, in English at any rate.
There is no need to use the plus (+) operator, String.concat or Array.join methods to consolidate strings together. TLs support interpolation through the ${variableName} syntax that can be used to combine content into a single string.
Finally, TLs recognise whitespace so there is no need for special syntax to split a text string over several lines; they support it out of the box.

"Whitespace", is a slightly ambiguous term, so in the context of this post, consider it to mean newline, tab and space characters.

There are a couple more “super-powers”: tagged templates and the raw property, but we will investigate those later.

Origins

The original idea for this post came from a talk given by Douglas Crockford on “The Better Parts” at JS Fest 2018. Around 22m15s in, Douglas describes how “megastring [template] literals” can be used to improve the understanding of Regular Expression patterns. There is a slight twist, as you will see, that it also uses a RegExp pattern to make the template literal digestible by the RegExp constructor, inside the “mega_regexp” function.

However, I suspect the code fragment might have been a late addition to the slide deck because it has a serious limitation in its whitespace removal. In fact the example given by Douglas in the talk does not actually work.

function mega_regxp(str, fl) {
    return new RegExp(str.replace(/\s/, ''), fl);
}

I have taken the idea several steps further and have used something similar in professional code.

Evolution

The first thing to do is correct the original code fragment so all the whitespace, added to aid formatting, is removed from the template literal by the function. This also means any whitespace you deliberately need in the pattern needs to be escaped. In fact there is a downside to this (initial) implementation in that special characters need to be double escaped, once within the template literal string and again within the RegExp.

function regExpTemplate(regExpString, regExpFlags = '') {
    return RegExp(regExpString.replaceAll(/\s+/g, ''),
        regExpFlags);
}

However, there is still room for improvement.

Utilising a Tagged Template

So far, the new function uses a Template Literal to help format the RegExp pattern, making it a little easier to understand. We are also able to interpolate values and sections of the pattern, which means we can apply meaningful names that further aid our understanding and provide some documentation.

There is another way to use TLs that opens up a potential improvement, and that is Tagged Templates, which is a special type of function intended to receive a TL decomposed into its (static) text sections and (interpolated) values. The function interface consists of at least one parameter which will be an array of the static text sections. There will always be one more section than interpolated values (even if the ends are empty strings), so the static sections start and finish the complete TL. The subsequence parameters are the interpolated values, which we will consolidate into an array using the rest parameter syntax.

We now have to revise our function to use this new super-power. First we curry the function so we can take an optional set of RegExp flags in our initial call and return a tagged template function. On the second call the tagged function receives the deconstructed sections of the TL, as described above. These have to be reconstituted to reform the complete TL. This is a stepping-stone to realising the super-power and may appear to be adding complexity for no tangible benefit.

function regExpTemplate(regExpFlags = '') {

    return (texts, ...values) => {
        const regExpString = texts.reduce(
            (pattern, text, index) =>
                `${pattern}${values[index - 1]}${text}`
        );

        return RegExp(regExpString.replaceAll(/\s+/g, ''),
            regExpFlags);
    };
}

Just in case you were wondering about the missing second argument of the reduce method, by default the first value of the source array is used in such cases. This is particularly helpful in this case because we are trying to interleaf the text and value sections to reform the original Template Literal, which are in arrays of different lengths.

Hear the TL raw

Next we will make use of the TL’s raw property and use the String.raw tagged template method to reconstitute the template string without the need to escape special characters.

function regExpTemplate(regExpFlags = '') {
    return ({ raw }, ...values) =>
        RegExp(
            String.raw({ raw },
                ...values).replaceAll(/\s+/g, ''),
            regExpFlags
        );
}

This revision simplifies our function again whilst enabling us to define the TL in a more natural form (without additional string escaping.)

Now it is nearly time to put our function through its paces, a little. This is not unit testing, just exercising the function with a few use case examples to demonstrate how it works and the benefits it brings. Before getting into the use cases we will first define some terminology and some helper functions to simplify the code.

Terminology

Validation: Confirmation that an item of data is a legitimate object in the domain. E.g. An email address is for a registered account.
Verification: Confirmation that an item of data confirms to a set of rules. E.g. The email address matches a Regular Expression pattern.

We can use RegExp to verify input conforms with a given pattern, but extending the pattern to perform validation has its limits. Even within those limits the resultant pattern is likely to become excessively convoluted and complicated.

Helper functions

In this context, helper functions are short (pure) self-contained functions used to simplify the performance of repetitive actions. The first we will define is used to perform an individual assertion and confirm the result is a 'PASS' or 'FAIL'.

function runTest(testRegExp, testString, expectedResult) {
    const actualResult = testRegExp.test(testString);
    console.log(
        `\t"${testString}"\tis expected to be ${expectedResult},  \twas actually ${actualResult} =\t${
            expectedResult === actualResult ? 'PASS' : 'FAIL'
        }`
    );
}

Next, ease the preparation of simple RegExp groups using:

const groupRegExp = (...options) => `(${options.join('|')})`;

This takes in a list of group values (options) and returns a string ready for use as a group in the RegExp pattern.

Finally, we enable the preparation of text sections containing escaped characters, without the need to double-escape them, using another String.raw tagged template method.

const escapeRegExp = ({ raw }) => String.raw({ raw });

It might also be worth defining the following constants to make it absolutely clear what is being formulated:

const FROM_START = '^';
const To_FINISH = '$';
const SPACE_CHARACTER = escapeRegExp`\s`;

However, we will not be using these initially so we can compare the initial TL against its simple String-based approach.

Some Use Case examples

Use Case scenario

The basic premise for the following use case is the need to confirm a person is between the ages of 18 and 79 as of a given date (1st Oct 2023 for test purposes). The person's date of birth is requested in 'dD MMM YYYY' format, where:

'dD' is single or double digit day of month, without leading zero.
'MMM' is a three letter English month with only the leading letter capitalised.
'YYYY' is a four digit year between 1900 and 2099, although this will be refined.
Each section of the date is separated with a single space.

The simplest pattern to verify the date format might look like /\d?\d [a-z]{3} \d{4}/i. Such a pattern would confirm '1 Oct 2005' is in the defined format but the pattern has loads of false positives. It includes unescaped space characters and is not limited to a complete string. I.e. '---00 xxX 9999---' would also match, so it need further refinement.

First we need to match complete strings so should prefix the pattern with ^ and suffix it with $.
Next, we should escape with space separators, replacing them with \s.
The first two digits of the year must be 19 or 20, so the format of the year should be (19|20)\d\d. However, we will improve on this later.
Months are a finite list of values so the following group will be suffice (Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec), and means we can remove the i flag used to ignore the case (uppercase/lowercase) of the text.
Lastly, the day of month is more complicated because we only want to permit values 1-31. Let's not worry about aligning with the selected month and leap-years; we will deal with that separately. There are three permutations to consider: 1-9, 10-29 and 30-31. So we can use the following pattern ([1-9]|[12]\d|3[01]).

It is considerations such as those listed above that make creating RegExp patterns so complicated in the first place, never mind testing them and maintaining them months later.

Our initial pattern would look something like this:

/^([1-9]|[12]\d|3[01])\s(Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)\s(19|20)\d\d$/

This already looks monstrous, and you are probably not viewing the line in its entirety. We have not yet ensured the given date in valid or truly represents a DoB within the 18-79 age range. For example, as of '3 Oct 2005' all of the following strings would verify correct.

'31 Feb 2000', which is an invalid date.
'4 Oct 2005' - '31 Dec 2005' would verify as 18 when they are still 17.
'1 Jan 1943' - '3 Oct 1943' would verify the person as being 79 when they have already had their 80th birthday.

In the last example (below) we will use an additional function to perform the final validation. This approach is often preferable to extending the RegExp pattern and avoids making it excessively complicated. However, before we employ such as function it is often necessary to perform the verification step so we know what input to expect.

Example One: hard coded

In each of the following three examples we will instantiate the tagged template in the same way using the regExpTemplate function and without RegExp flags.

const testRegExpTag = regExpTemplate();

Then we create the testRegExp object using the a Template Literal distributed over 7 lines. This makes it far easier to see the individual sections when compared to the conventional String approach.

const testRegExp = testRegExpTag`
^
    ([1-9]|[12]\d|3[01])
    \s
    (Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)
    \s
    (19|20)\d\d
$`;

We can exercise testRegExp using the following test cases.

runTest(testRegExp, '1 Jan 1900', true);
runTest(testRegExp, '20 Feb 2000', true);
runTest(testRegExp, '31 Dec 2099', true);

runTest(testRegExp, '31 Dec 1899', false);
runTest(testRegExp, '1 Xxx 2000', false);
runTest(testRegExp, '1 Jan 2100', false);

The above test cases confirm valid strings pass and malformed/out of bounds strings fail as expected. However, we can make the pattern even more maintainable and testable.

Example Two: interpolated

We will commence this example by defining the following constants using the helper functions we defined earlier.

const DAY_OF_MONTH = groupRegExp('[1-9]',
    escapeRegExp`[12]\d`, '3[01]');

const MONTHS_OF_YEAR = groupRegExp(
    'Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun',
    'Jul', 'Aug', 'Sep', 'Oct', 'Nov', 'Dec');

const YEAR_RANGE = escapeRegExp`(19|20)\d\d`;

Using the above constants (including those we defined with the helper functions), we can replace the 7 lines of code from the previous example as follows.

const testRegExp = testRegExpTag`
    ${FROM_START}
    ${DAY_OF_MONTH}
    ${SPACE_CHARACTER}
    ${MONTHS_OF_YEAR}
    ${SPACE_CHARACTER}
    ${YEAR_RANGE}
    ${To_FINISH}
`;

I hope you agree, this is far more self-documenting. Imagine how much easier it would be to replace the space separation with a hyphen ('-') given the above definition.

Example Three: restricted

We can improve the format verification a little by extending the year definition to range for 18-79 year olds (as of 2023). We can extend the RegExp pattern to perform even more date validation but this would make the pattern extremely complicated, so we will employ a JS function for validation. This will require use of an array of strings for the months so we will redefine the MONTH_OF_YEAR section to use a string array.

const MONTHS_ARRAY = ['Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun',
    'Jul', 'Aug', 'Sep', 'Oct', 'Nov', 'Dec'];

const MONTHS_OF_YEAR = groupRegExp(...MONTHS_ARRAY);

const YEAR_RANGE = 
    escapeRegExp`((19(4[3-9]|[5-9]\d))|(200[0-5]))`;

Our test cases now confirm we can trap DoBs for those who will turn 18 or 80 this year.

runTest(testRegExp, '1 Jan 1943', true);
runTest(testRegExp, '31 Dec 2005', true);
runTest(testRegExp, '31 Dec 1942', false);
runTest(testRegExp, '1 Jan 2006', false);

To perform validation of the user input date we need to consider two stages:
1) Validation of the input as a valid date, which assumes the input passed format verification.

We know the day will be numeric between 1 and 31,
We know the month will be one of the twelve values in MONTHS_ARRAY, and
We know the year will be a numeric between 1943 to 2005, to cover possible years for 18 to 79 year olds.

In the above list of assertions, the day and year sections were described as numeric because they will actually be strings and will need conversion to numbers; we will use the + prefix to achieve this.

2) Validate the DoB is with in the 18-79 age range as of today. Note, the value of today is a parameter with a default value of the current Date. This makes it possible to override the value of today when called, which makes testing considerably simpler.

function validateDoB(verifiedDobString, today = new Date()) {
    const [dobDay, dobMonth, dobYear] = verifiedDobString.
        split(' ');
    const monthNum = MONTHS_ARRAY.indexOf(dobMonth);
    const dobDate = new Date(+dobYear, monthNum, +dobDay);
    const dob18 = new Date(
        today.getFullYear() - 18,
        today.getMonth(),
        today.getDate()
    );
    const dob80 = new Date(
        today.getFullYear() - 80,
        today.getMonth(),
        today.getDate()
    );
    return (
        +dobYear === dobDate.getFullYear() &&
        monthNum === dobDate.getMonth() &&
        +dobDay === dobDate.getDate() &&
        dobDate.valueOf() <= dob18.valueOf() &&
        dobDate.valueOf() > dob80.valueOf()
    );
}

Conclusion

Building on Douglas Crockford's original concept we can create a simple mechanism for preparing RegExp patterns that are easier to understand and maintain, and should be easier to test. The approach extends from the simple Template Literal approach that enables use format the pattern with additional whitespace. Using the TL we can also construct the pattern with named values to improve documentation.

Some guidance for using RegExp

My ~~four~~ five rules of best practice for Regular Expressions:

Whenever possible top and tail the pattern. It is not always possible but you should always consider marking the beginning of the pattern with the caret symbol ^ and the end of the pattern with the dollar symbol $ to ensure full text, initial text or end text matches.
Try to bound repetition. Instead of using + for 1 or more, and * for 0 or more, consider what might be a reasonable upper limit and use the range notation {L, U}, where L is the lower limit (0 or 1 in most cases) and U is the upper limit. However, all of the above are greedy, which means they will match all they can. This can be reduced/optimised by following the repetition syntax with ? so the repetition will conclude with the first complete matching pattern.
Test, test and test some more:
- Test a selection of the cases you expect it to match.
- Test the edge cases (+ve and -ve) to confirm the boundary.
- Test as many exceptions to the rule you can identify to ensure false positives are detected early.
Whatever your position on commenting code, I think documenting what the author was intending to achieve with a RegExp pattern is usually a good idea.
In light of the Cloudflare RegExp Outage in July 2019, care should be taken when matching around a delimiting character. Using the .* (any number of any character) pattern is rarely a good idea. Consider limiting the type of characters to those expected, i.e. Exclude the delimiter character itself.

In the Cloudflare incident, the end of the RegExp pattern included .*(?:.*=.*). Excluding the non-capturing group results in the pattern .*.*=.*, which given the greedy nature of RegExp pattern matching, is a rather ravenous little beasty. The following changes might have been an improvement.

Remove one of the leading .* patterns as it is obsolete.
Replace the leading any-character search with something more specific such as [^=]*, where the delimiting equals symbol is excluded.

Here is more on the Cloudflare incident.

Some additional advice when in JS:

Use the RegExp.exec method in preference to String.match, apparently it is faster.
Beware of building too much logic into the pattern. This can make the pattern excessively complicated and there are often a better way to implement the logic.
Study the documentation as each implementation has its quirks, even when based on POSIX.
Use a visualisation tool to gain insight into the structure of you pattern, A good resource for this is Regulex.

Streaming SIMD in JS

Tracy Gilmore — Sun, 20 Aug 2023 15:23:43 +0000

In my previous post on this topic I described how we could use the concept behind SIMD to enhance JS performance. The implementation was limited to using a single data array/vector. In the implementation that follows, I have taken inspiration from Streaming SIMD Extensions (SSE) to extend my implementation to support multiple data streams.

Things to know before reading this post.

Quick recap of sim-simd

My simd() function takes a regular function as its only parameter and returns a new function instrumented for SIMD operation. The simd-version of the function returned, expects one or more homogeneous (all of the same data type) arguments. When called, the original function will be applied to each parameter (using a map method) but with the execution performed inside their own Promise to leverage the benefit of concurrent execution. The allSettled Promise method is used to collate the results, that are returned from the function, in an array.

export default function simd(instruction) {
  return function (...data) {
    const executions = data.map(
      datum =>
        new Promise((resolve, reject) => {
          try {
            resolve(instruction(datum));
          } catch (error) {
            reject(error);
          }
        })
    );
    return Promise.allSettled(executions);
  };
}

NB, the execution interface will be changing to support multiple data feeds. Although they are different things, the 'data feeds' can be streams, vectors or regular arrays in the case of JS.

Enhanced SIMD (esimd)

Being able to process multiple data feeds at the same time raises a significant question but also offers a considerable opportunity.

What does the function do if the data feeds (arrays) are of different lengths?

In my mind there are three options, which we will call no-cache, cached and matrix execution modes.

export const ExecutionMode = {
  NO_CACHE: 0,
  CACHED: 1,
  MATRIX: 2,
};

No-cache is by far the simplest as we only use what data we can and disregard the rest. This might seem wasteful but in time critical applications this might be the best option.

Cached retains any unused values to prepend them to the beginning of the next wave of data. Data is not lost but there has to be an upper limit and a way to clear the cache.

Finally we have the Matrix execution mode, which could be very effective. In this mode all the data is used because the original function is applied with every combination of input values.

For example, if the original function expects two parameters and its esimd version is presented with [2, 3], [4, 5], the original function would be called 4 times in total with the following arguments: (2, 4), (2, 5), (3, 4) & (3, 5). If the original function was a simple multiplication, its esimd function would return the following 2-dimensional array: [[8, 10], [12, 15]]. Similar to its simpler cousin, all four operations will be executed in concurrently courtesy of Promises.

Exceptions

The number of data feeds needs to match the number of parameters expected by the original function, otherwise an exception will be thrown. If one of the executions of the instruction fails (status !== "fulfilled") an exception will also be raised, but if execution of the instruction results in an exception the return status will be "rejected" and the value undefined.

In matrix mode there are three additional exceptions to limit resource consumption from:

too many dimensions stipulated for the matrix (default max 10)
too large a dimension stipulated for the matrix (default max 10)
too great a resultant matrix (default max 10,000 values)

Testing

As illustrated by the graphic at the head of this post, we will be exercising each of the three modes described above by passing in an instruction (function) that excepts three inputs (characters) and returns the values concatenated into a string. However, after converting the instruction process into an enhanced-simd (esimd) version (esimdInstruction) we will pass in the data as arrays of characters. The test instruction is defined in such a way that, should the concatenated string be '666' an exception will be thrown.

The test instruction and data

// process.js (CJS module)
async function process(a,b,c) {
  const result = `${a}${b}${c}`;
  if (result === '666') {
      throw Error('Number of the beast');
  }
  return result;
}

// Un-exceptional test data (imported JSON)
[
  ["A", "B"],
  ["C", "D", "E"],
  ["F", "0", "1", "2"]
]

// Exceptional test data
const testData = [
    ["A", "6", "B"],
    ["C", "6", "D", "E"],
    ["F", "6", "0", "1", "2"]
];

Reference the unit tests in my GitHub repository so see how these were used to exercise the source code.

Implementation

The following section will demonstrate the construction of the function to deliver each of the three modes described above, individually. We will conclude the section by combining the code to manage all three modes on demand. In preparation we will start by reviewing some of the supplementary code.

Supplementary code

The following four code fragments are used, in combination, at the core of the three modes. Extracting these functions will greatly simplify the functions fro each mode.

executeInstruction

If the following fragment of code appears familiar, it should. It is at the heart of the simd implementation and is responsible for converting an instruction (fnInstruction) and a dataset (array) argument into a Promise.resolve, or a Promise.reject should the function fail (throw an exception).

function executeInstruction(fnInstruction) {
  return (dataset) =>
    new Promise(async (resolve, reject) => {
      try {
        resolve(await fnInstruction(...dataset));
      } catch (error) {
        reject(error);
      }
    });
}

The above function is used by all three modes to perform the operation and generate the Promise.

transform

Both the no-cache and cached modes need to convert the data before it is processed. Instead of an array for each argument, we need a set of arrays, each containing a value for each argument.

For example, instead of the two arrays containing the data feeds ['A', 'B'] and ['C', 'D'] where each array contains the values for a specific argument, we want ['A', 'C'] and ['B', 'D'] so each array contains a single value for each argument. This rotation of the arrays are performed by the transform function, with the swapRowColumn (sub)function performing the replacement of rows for columns.

function transform(fn, dataSources, caches) {
  const swapRowColumn = (_, row) =>
    row.map((__, i) => [...(_[i] || []), row[i]]);

  return cacheSurplus(caches, dataSources)
    .reduce(swapRowColumn, [])
    .map(executeInstruction(fn));
}

The cacheSurplus function used in transform is described next.

cacheSurplus

This function performs two operations, which I accept conflicts with the Single Responsibility principle, but the two operations are closely related.

First we need to extract the groups of data from the input to feed the arguments of the instruction. The second operation is to preserve any excess data in a cache and prefix it to the next feed, where required (caches is populated).

function cacheSurplus(caches, dataFeeds) {
  const dataFeedSize = dataFeeds.reduce(
    (feedSize, dataFeed, index) => {
      if (caches) dataFeed.unshift(...caches[index]);
      return Math.min(feedSize, dataFeed.length);
    },
    Infinity
  );

  const newCache = dataFeeds.map((dataFeed) =>
    dataFeed.splice(dataFeedSize));

  if (caches) {
    caches.length = 0;
    caches.push(...newCache);
  }
  return dataFeeds;
}

permute

The permute function contains a recursive function _permute that generates a multi-dimensional array of Promises. The array is flattend into a single dimensional array so it can be used with the Promise.allSettled method.

By multi-dimensional array I mean a set of nested arrays. One dimension has no nesting and can be imagined as a simple list. Two dimensions (2D) form an array of arrays, that can be visualised as a plain, rectangle or grid. A 3D array is an array of arrays of arrays, and is analogous to a solid-box, or array of grids. Trying to visualise more than 3 dimensions gets rather difficult, but by now you probably see the pattern.

function permute(fn, dataFeeds) {
  const dataFeedLength = dataFeeds.length;
  return _permute().flat(dataFeedLength - 1);

  function _permute(...buffers) {
    const buffersLength = buffers.length;
    return buffersLength === dataFeedLength
      ? executeInstruction(fn)(buffers)
      : dataFeeds[buffersLength].map((dataFeed) =>
          _permute(...buffers, dataFeed)
        );
  }
}

Notice that the permute function calls the executeInstruction indirectly using partial-application, where a call to executeInstruction with the instruction function (fn) to create a new function executeFn.

No-Cache mode

In my preparation for this post I decided to develop the code for each mode individually although the cached mode is built on the no-cache mode because they are so similar.

function esimd(instruction) {
  return async function (...dataSources) {
    const executions = transform(instruction,
      structuredClone(dataSources));

    const results = await Promise.allSettled(executions);
    return results.map((result) => result.value);
  };
}

The no-cache example is relatively simple. Like all of the esimd modes, it started by taking in a function that is the instruction we want to perform of the data sets, and returns a function in a typical closure pattern. The 'esimd'-ised version of the instruction is based on Promises so operates _async_hronously and is expected to be called with the name number of array arguments as the number instruction parameters.

The executions variable is the result of transforming the input arrays (one per parameter) into a collection of arrays, one per execution containing a single set of arguments. The structuredClone function is new to JS and is used to duplicate the data so the original source is not mutated by esimd. The transform function does more than rearrange the values into argument sets, it also executes to instruction that generates a Promise, the result being an new array of Promises for each complete set of arguments.

In the no-cache version of esimd any data that cannot be used to form a complete argument set is discarded. We then await the results to be settled using the Promise.allSettled method and extract the values to return to the caller. There are a number of exceptions that we can manage, as described above, but we will leave discussing those for now and only consider the 'happy path'.

Cached mode

The cached mode builds on the no-cache version with the main difference being we include a caches array (of arrays) to retain any data that did not form part of a complete argument set. The caches is passed into the transform function where that values can be prefixed to the new data feeds.

function esimd(instruction) {
  let caches = [...Array(instruction.length)].map((_) => []);

  return async function (...dataSources) {
    const executions = transform(
      instruction,
      structuredClone(dataSources),
      caches
    );

    const results = await Promise.allSettled(executions);
    return results.map((result) => result.value);
  };
}

Matrix mode

Supporting matrix mode is very different to the previous two modes. Here we try to use very permutation (permute) of the argument arrays, as described much earlier. Otherwise the actual esimd function is extremely simple.

function esimd(instruction) {
  return async function (...dataSources) {
    const executions = permute(instruction,
      structuredClone(dataSources));

    const results = await Promise.allSettled(executions);
    return results.map((result) => result.value);
  };
}

Bringing it all together

With a single function to manage all three modes we need a way to inform the function which mode to use. This is indicated by the second parameter executionMode of the initial call, which (is optional, defaulting to no-cache) uses one of three values provided by the ExecutionMode object (pseudo-enumeration). We, optionally, initialise a caches array depending on the mode and execute either the permute or transform function, again depending on the mode of execution.

function esimd(
  instruction, 
  executionMode = ExecutionMode.NO_CACHE
) {
  const caches =
    executionMode === ExecutionMode.CACHED
      ? [...Array(instruction.length)].map((_) => [])
      : null;

  return async function (...dataSources) {
    checkAlignment(instruction, dataSources);
    checkCapacity(executionMode, dataSources);

    const executions = (
      executionMode === ExecutionMode.MATRIX 
        ? permute : transform
    )(instruction, structuredClone(dataSources), caches);

    const results = await Promise.allSettled(executions);
    validateResults(results);
    return results.map((result) => result.value);
  };
}

In this final version we also include some protection against anticipated exception conditions:

checkAlignment ensures the number of data sources matches the number of parameters of the instruction function.
checkCapacity attempts to protect the computer from managing matrix operations that are too large in terms of source data or the expected result output.
validateResults flags up when any of the instruction executions resulted in an error/exception.

The source code for this post can be found in my GitHub repository. However, it is not up to production grade and should only be used for educational purposes.

Besides the obvious support for multiple data feeds, there are two significant differences between enhanced-simd and sim-simd.

The simulated edition returns the result of the Promise.allSettled in its unrefined form, which might be confusing for some developers. Instead of an array of return values, they will receive an array of objects containing properties of:
- status ('fulfilled' or 'rejected') indicating if the function was successful or not.
- value is the actual return value from the function call.
- reason is included only if the function was 'rejected' and states the cause of the failure.
To protect the developer and the system, there is far more use of exception handling and reporting. We need to make sure that, if the input provided is going to impact system performance, we are aware of it prior to it affecting end users. We, as developers, can then make a determination as to how best to manage it.

Another minor difference between the two implementations is that simd is provided as an ES6 module but esimd is coded using the Common JS module approach. It is a minor issue to change from CJS to ESM but the unit testings is a little more complicated and adds nothing to the topic at hand.

I have included unit tests in the GitHub repo to aid understanding on how esimd can be used. The unit tests include several that exercise the asynchronous function such that it throws an exception. I found this unit test quite challenging, but this Dev.to post from Alessio Michelini was exceptional useful.

Update April 1st, 2024 (no fooling)

In response to a comment I received on this topic, I had previously confused the term parallel with concurrent in the context of processing models. I have, hopefully, now corrected the error in this and the previous article but it also raises an interesting point for this article.

Whilst JS is, by default, single threaded, especially in the browser where it is primarily concerned with updating the DOM and responding to user actions, there is a Web API called Web Workers that can provide additional threads of execution.

Web Worker enable computationally intensive activity to be conducted away from the main thread, which reduces the potential impact on the user experience by having the browser blocking interaction. I would advise that the architecture of an application should first try to relocate such processing to the server but this might be a reasonable trade-off in some circumstances.

In order to make use of Web Workers in a SIMD fashion, it would be necessary to replace the function parameter with the URL of a JS script, prepared in a particular way. The SIMD wrapper could then fame out the work over a number of threads. This would utilise more than one processor on the machine, adopting a more parallel module of execution rather than concurrent.

If you would like an article (and code) demonstrating how this could be achieved, please leave a comment below.

Exploring DO Notation in JS

Tracy Gilmore — Mon, 07 Aug 2023 12:35:54 +0000

Introduction

In the pure FP language Haskell (and others) there is a composing mechanism known as “Do Notation”. In this post we will be mimicking this mechanism using JavaScript, not to poke fun at it but to investigate it and add it to our solution toolkit.

The Do Notation is typically used to combine Monads to perform operations such as IO (Input/Output), but we will not be discussing that subject here. It is sufficient to know the following:

Monads are often used to isolate operations that result in side-effects in order to protect the rest of the application from unexpected consequences.
Monads behave like a simple (pure) function, which is what we will be using.
The functions return a value from an operation that expects to be supplied with a single value; although the value could be complex (object and/or array).

There is a similar approach for combining operations called pipelines. See the footnote to discover more about the TC39 proposal to bring pipes to JS. In both cases the composition takes the form of a chain of operations with the output of one being the input of the next.

The use case

In order to demonstrate and contrast the two approaches we could do with a simple process to exercise the techniques. We will be using how we can convert temperatures between degrees Celsius (C) and degrees Fahrenheit (F). If you are familiar with this process feel free to skip this section. If you are not, and especially if you are uncomfortable with mathematics, I promise to take you through the calculations slowly.

Converting temperatures between C and F

Believe it or not, the mathematics for this is simple and only uses the sort of operations found on a basic calculator: addition (+), subtraction(-), multiplication(x, although in computing we use the symbol *) and division (/).

The two scales, C and F, are consistent and can be represented as a straight line on a graph. Keep with me and I will explain.

If we take water, freeze it solid and measure its temperature we can get two numbers. When the thermometer is in F mode we get 32, when in C mode we get zero. If we boil water and measure its temperature we again get two values: 212°F and 100°C. This is represented on the above graph as the red line. The F values run up and down the left (vertical) axis and the C scale runs along the bottom horizontal axis.

We can use the graph to convert Fahrenheit to Celsius by simply finding the temperature on the left axis and following a path horizontally to the red diagonal line. At the point we hit the line, trace a vertical path down to the bottom axis where we will find the Celsius temperature. Our functions will do something similar to this process.

If we adjust the F temperature by subtracting 32, making frozen water also 0F, the diagonal line now passes through 0 on both axes. Shown as the green line but notice the red and green lines remain parallel, like a pair of train tracks, the distance between them remains the same from one end to the other and they never cross.

However, boiling water would become 180°F but still 100°C, the slope of the diagonal line remains the same. This is important because it means as C changes F also changes, not by the same amount but by a consistent rate or ratio. In fact as C goes from freezing (0) to boiling (100), the temperature in F increases by 180. At 50°C the point on the green line (F - 32) is 90. To get the actual temperature in F (red line) we need only add the 32 back on = 90 + 32 = 112. The relationship of the slope is 100:180, which is also 50:90 (as shown above) and 5:9. For every increase in C by 5 degrees, F will increase by 9 degrees and this is consistent.

Now for the formulas/equations (recipes) for C to F and F to C, with examples, which is just another way of showing what we found out above.

((C * 9) / 5) + 32 gives us F

((F - 32) * 5) / 9 gives us C

Let’s convert Celsius to Fahrenheit, and back again

Boiling point: 100°C * 9 = 900
    900 / 5 = 180
        180 + 32 = 212°F

Freezing point: 0°C * 9 = 0
    0 / 5 = 0
        0 + 32 = 32°F

Now for converting F to C,

Boiling point: 212°F - 32 = 180
    180 * 5 = 900
        900 / 9 = 100°C

Freezing point: 32°F - 32 = 0
    0 * 5 = 0
        0 / 9 = 0°C

There is a third number on the far left end of the red line that is interesting. What makes it interesting is, it is the same value in C and F, that is -40degrees.

-40°C * 9 = -360
    -360 / 5 = -62
        -62 + 32 = -40°F

Also,

-40°F - 32 = -72
    -72 * 5 = -360
        -360 / 9 = -40°C

In all of the calculations above we used only four simple operations:

Addition (+ 32)
Subtraction (- 32)
Multiply (* 9 and * 5)
Divide (/ 5 and / 9)

Combining three of these operations in just the right order (as illustrated above) will provide us with the calculations we need. If each operation is a simple function, combining them is the same as composing the functions to create a temperature conversion function.

The Canonical imperative approach

It is relatively straightforward to code the above calculations in the imperative (function/method) approach. In fact some of the brackets in the code could be removed but have been included to align with the above explanation and later code.

/* canonical.js */

import runTestCases from './testcases.js';

const INTERCEPT = 32;
const CELSIUS_DELTA = 5;
const FAHRENHEIT_DELTA = 9;
const SLOPE = CELSIUS_DELTA / FAHRENHEIT_DELTA;

function celsiusToFahrenheit(c) {
    return c / SLOPE + INTERCEPT;
}
function fahrenheitToCelsius(f) {
    return (f - INTERCEPT) * SLOPE;
}

runTestCases(celsiusToFahrenheit, fahrenheitToCelsius);

Notice the import at the top of the above code fragment. It is used to bring in a testing capability, which is exercised on the last line by calling the runTestCases function. In this example it will present the two temperature functions with the following test cases:

/* Fragment of the testcases.js file (testcase 1) */

celsiusToFahrenheit: [
    { input: 100, expected: 212 },
    { input: 0, expected: 32 },
    { input: -40, expected: -40 },
],
fahrenheitToCelsius: [
    { input: 212, expected: 100 },
    { input: 32, expected: 0 },
    { input: -40, expected: -40 },
],

The tests call the function using the input value and then compares the output against the expected value. Here are the results.

In the next example we continue in the imperative coding style but in a more elaborate way en-route to the subject of this post.

/* imperative.js */

import runTestCases from './testcases.js';

function add(m, n) {
    return m + n;
}
function sub(m, n) {
    return m - n;
}
function mul(m, n) {
    return m * n;
}
function div(m, n) {
    return m / n;
}

function celsiusToFahrenheit(c) {
    return add(div(mul(c, 9), 5), 32);
}
function fahrenheitToCelsius(f) {
    return div(mul(sub(f, 32), 5), 9);
}

runTestCases(celsiusToFahrenheit, fahrenheitToCelsius);

We exercise the functions in the same way as before and they produce the same results. The big difference here though is the way the mathematical operations are prepared and used. Notice how the div and mul functions are used twice. Also notice how the temperature conversion functions are composed of simple mathematical operations.

Going functional

The next example adopts a more 'functional' style of coding, as in Functional Programming, instead of the procedural and other imperative style demonstrated so far.

/* functional.js */

import runTestCases from './testcases.js';

import { specificOperations } from './operations.js';

const { add32, div5, mul9, div9, mul5, sub32 } = specificOperations;

// Conversion functions
function celsiusToFahrenheit(c) {
    return add32(div5(mul9(c)));
}
function fahrenheitToCelsius(f) {
    return div9(mul5(sub32(f)));
}

runTestCases(celsiusToFahrenheit, fahrenheitToCelsius);

Testing of the above code fragment is exactly the same as before but this time we are also importing an object containing specific operations from the operations module (described below). These functions make it more obvious what is being done and reduce attention to how it is being done, which is the declarative nature of Functional Programming (FP). However, the way we compose the operations to for the conversion functions remains more imperative than declarative.

Generic and Specific Operations

The mathematical functions that underpin the operations are very simple but function composition would typically be more involved.

/* operations.js */

// Generic operations
function addM(m) {
    return n => n + m;
}
function subM(m) {
    return n => n - m;
}
function mulM(m) {
    return n => n * m;
}
function divM(m) {
    return n => n / m;
}

export const genericOperations = {
    addM,
    subM,
    mulM,
    divM,
};

// Specific operations
const add32 = addM(32);
const div5 = divM(5);
const mul9 = mulM(9);
const div9 = divM(9);
const mul5 = mulM(5);
const sub32 = subM(32);

export const specificOperations = {
    add32,
    div5,
    mul9,
    div9,
    mul5,
    sub32,
};

In the above module two objects are exported genericOperations, specificOperations, with the specificOperations based on the genericOperations. The genericOperations are based on the four basic mathematical functions using the FP technique known as currying so the parameters can be supplied independently. This enables the specificOperations to employ another FP technique known as partial application where the first argument is provided (bound to the first parameter) to generate a specialised function (operation). See this article for a more complete explanation of the techniques.

Do Notation in JS

In some FP languages the Do Notation is “idiomatic” meaning it is built into the language. Although JS comes with some features taken from the FP school, Do Notation is not one of them, so we have to recreate it ourselves. There is some new syntax in the "pipeline" but it is a little way off yet (see footnote 2).

In the next seven code examples we will be developing a set of functions to simulate Do Notation; exploring fragments from the do-notation.js module as we go.

"Cracker" diagrams

As an alternative way of describing each of the following examples, I have devised a way of illustrating the functionality that bears an uncanny resemblance to Christmas Crackers. If you are not familiar with the novelty, you might find this Wikipedia page of interest.

The diagrams flow from left to right and employ the following symbols.

The diagrams attempt to show how data supplied to the composed function (at the left) flows through a series of functions to produce the output value (at the right).

Example 1: DOing it with specific functions

The 'cracker' diagram above is a depiction of the two DO compositions below. Observer how they take in a numeric value, pass through a series of three specific functions (composed into a single DO function) and return a numeric value as output. The previous sentence is true for both the diagram and the source code.

/* do-mk1.js */

import runTestCases from './testcases.js';

import { DO } from './do-notation.js';

import { specificOperations } from './operations.js';

const { add32, div5, mul9, div9, mul5, sub32 } = specificOperations;

// Conversion functions
const celsiusToFahrenheit = DO(mul9, div5, add32);

const fahrenheitToCelsiusOperations = [sub32, mul5, div9];
const fahrenheitToCelsius = DO(fahrenheitToCelsiusOperations);

runTestCases(celsiusToFahrenheit, fahrenheitToCelsius);

In the above code fragment the specific functions are composed using the DO function. These will operate on an initial numeric input to produce a numeric output but we will expand on this later, but first we will define the DO function.

/* Fragment of do-notation.js */

export function DO(...fns) {
    return data => 
        fns.flat().reduce((output, fn) =>
            fn(output), data);
}

The DO function is quite simple. It uses the rest syntax to accept a list of functions as a single array parameter. It returns a function that expects a single data parameter. When called, the returned function 'pipes' the input from one function to another, using the reduce method, with the final result being the output of the DO function.

Example 2: DOing it with generic functions

This example is virtually identical to the previous but uses the genericOperations. This means there are fewer operations imported but when called they have to be supplied with the initial argument to specilise them.

/* Fragment of do-mk2.js */

const { addM, subM, mulM, divM } = genericOperations;

// Conversion functions
const celsiusToFahrenheit = DO(
    mulM(9), 
    divM(5), 
    addM(32)
);

const fahrenheitToCelsiusOperations = [
    subM(32),
    mulM(5),
    divM(9)
];
const fahrenheitToCelsius = DO(fahrenheitToCelsiusOperations);

Example 3: Using formatted (string) input

In the third example we will use a string to represent the input temperature but this is just a stepping-stone so our output will still be numeric. The primary purpose of this example is to demonstrate the type of data input can be different from that of the output.

/* Fragment of the testcases.js file (testcase 2) */

celsiusToFahrenheit: [
    { input: '100°C', expected: 212 },
    { input: '0°C', expected: 32 },
    { input: '-40°C', expected: -40 },
],
fahrenheitToCelsius: [
    { input: '212°F', expected: 100 },
    { input: '32°F', expected: 0 },
    { input: '-40°F', expected: -40 },
],

The test results report is slightly different.

Notice the input value is a string that combines the numeric value and the scale (C or F) separated by the degree symbol (°). This means the numeric value will need to be extracted from the input string as part of the DO composition using the following function.

function extractTemp(tempStr) {
    return parseInt(tempStr, 10);
}

The conversion function will be constructed as follows:

/* Fragment of do-mk3.js */

// Conversion functions
const celsiusToFahrenheit = DO(
    extractTemp, 
    mulM(9), 
    divM(5), 
    addM(32)
);

const fahrenheitToCelsiusOperations = [
    extractTemp,
    subM(32),
    mulM(5),
    divM(9),
];

const fahrenheitToCelsius = DO(fahrenheitToCelsiusOperations);

Notice the combination of function references and function calls (that return a function reference) in the list of functions of the DO instruction. Ideally, we also like the output to be a string so on the example four.

Example 4: Conditional processing of formatted input & output

In this example our conversion functions will take in data as a string and produce the output as a string.

/* Fragment of the testcases.js file (testcase 3) */

celsiusToFahrenheit: [
    { input: '100°C', expected: '212°F' },
    { input: '0°C', expected: '32°F' },
    { input: '-40°C', expected: '-40°F' },
],
fahrenheitToCelsius: [
    { input: '212°F', expected: '100°C' },
    { input: '32°F', expected: '0°C' },
    { input: '-40°F', expected: '-40°C' },
],

At this point we will introduce a mechanism for conditional execution (a.k.a. branching) through the IF function in conjunction with the isCelsius predicate function defined below.

/* Fragment of do-mk4.js */

function isCelsius(tempStr) {
    return tempStr.at(-1).toUpperCase() === 'C';
}
function convertToString(scale) {
    return n => `${n}°${scale.toUpperCase()}`;
}

The compositions now look like this.

// Conversion functions
const celsiusToFahrenheit = [
    extractTemp,
    mulM(9),
    divM(5),
    addM(32),
    convertToString('F'),
];

const fahrenheitToCelsius = [
    extractTemp,
    subM(32),
    mulM(5),
    divM(9),
    convertToString('C'),
];

const convertTemperature = DO(
    IF(isCelsius,
        DO(celsiusToFahrenheit),
        DO(fahrenheitToCelsius)
    )
);

The above code wraps the IF function in a DO operation but as this is the only task in the composition, and the IF call returns a function, it could be executed directly. The do-notation IF function is very simple, defined as follows.

export function IF(condition, doTrue, doFalse) {
    return data => (condition(data)
        ? doTrue(data)
        : doFalse(data)
    );
}

The above function accepts three parameters, all functions, and returns a new function that takes a single input as part of the DO composition. The first 'condition' is what is known as a predicate function because it converts its input into a Boolean output (true or false). This decides which of the next two functions will be executed. In our example, the condition identifies the scale of the input (C = true, or F = false) and performs the appropriate doTrue for celsiusToFahrenheit or doFalse for fahrenheitToCelsius.

Example 5: DO_IF_THEN_ELSE

The do notation IF function is a little less readable than its imperative equivalent so in this example we make use of function chaining to make a more "readable" mechanism.

export function DO_IF(condition) {
    return {
        THEN_DO: doTrue => ({
            ELSE_DO: doFalse => 
                data => DO(condition(data)
                    ? doTrue 
                    : doFalse)(data),
        }),
    };
}

The implementation is a little more complicated, which is often the trade-off, but in exchange we abstract the complexity from where we want to use it.

/* Fragment of do-mk5.js */

const convertTemperature = DO_IF(isCelsius)
    .THEN_DO(celsiusToFahrenheit)
    .ELSE_DO(fahrenheitToCelsius);

At this point which approach to conditionals, I think, is a matter of personal preference as they have the same effect. Another common processing mechanism we might want to employ is loops, so here is our next example.

Example 6: `DO_WITH`

In this example we want to process multiple input values in a single call. Here are the test cases presented as properties of an object we will be supplying to the DO composition. The name of each property is the input value of a single calculation, the property value is the expected output for comparison.

/* Fragment of the testcases.js file (testcase 5) */

{
    '100°C': '212°F',
    '0°C': '32°F',
    '-40°C': '-40°F',
    '212°F': '100°C',
    '32°F': '0°C',
    '-40°F': '-40°C',
}

After passing the above values through the DO process we get the following results.

For comparison, here is the source code.

/* Fragment of do-mk6.js */

const extractInputs = _ =>
    Object.entries(_).map(([input, expected]) => ({
        input,
        expected,
    }));

const convertInput = _ => ({ ..._, actual: 

convertTemperature(_.input) });

const evaluateResult = _ =>
    ({ ..._, result: _.expected === _.actual });

console.table(
    DO_WITH(
        extractInputs, 
        convertInput, 
        IDENTITY, 
        evaluateResult
    )(testCases[4])
);

Example 7: The finale - Object processing

Here we are with the final example in which we process complex data held in an object. To do this we will be employing more of a Monad-ic style through the processObject function that will be used to wrap the partially applied, mathematical operations supplied by the genericOperations object. The wrapper is used as an adaptor that makes the input data object into something the specialised functions can work with. It also takes the calculated result and converts it back into another object, ready for the next call.

function processObject(func) {
    return tempObj => ({
        num: func(tempObj.num),
        scale: tempObj.scale,
    });
}

function extractTemp(tempStr) {
    const [temp, scale] = tempStr.split(/°/);
    return { num: +temp, scale };
}

function convertToString(tempObj) {
    const newScale = isCelsius(tempObj) ? 'F' : 'C';
    return `${tempObj.num}°${newScale}`;
}

The extractTemp function converts the test case string into the object we will be passing through the DO process. Conversely, the convertToString will encode the finished object back into a string for validation and presentation. However, the conversion functions do look a little odd with all the processObject wrappers.

// Conversion functions
const celsiusToFahrenheit = DO(
    processObject(mulM(9)),
    processObject(divM(5)),
    processObject(addM(32))
);

const fahrenheitToCelsius = DO(
    processObject(subM(32)),
    processObject(mulM(5)),
    processObject(divM(9))
);

But the convertTemperature function does not look vastly different from that in example four.

const convertTemperature = DO(
    extractTemp,
    IF(isCelsius,
        celsiusToFahrenheit, 
        fahrenheitToCelsius),
    convertToString
);

Errata

As identified by @javarobit in the comments below, for the isCelsius function to work with the IF construct, its implementation has to change to something like:

function isCelsius(tempObj) {
    return tempObj.scale.toUpperCase() === 'C';
}

Thanks again to @javarobit.

In Summary

In my mind, and I am sure it has been said by others, Functional Programming is all about composition. Combining smaller (and simpler) functions together into a larger, more complicated (and usually dedicated) function so they execute all at the same time.

JavaScript is gradually acquiring FP-style capability that can greatly enhance your tool kit. You don't have to use FP or OOP entirely. A developer should strive to use the most appropriate tools to formulate the solution required for the problem at hand. I have used FP and OOP features in combination to great effect and produced solutions that are easier to understand, test and maintain.

Footnotes

It has to be said that the Do Notation in Haskell is not universally loved, as indicated in this page of the Haskell documentation.
There is an ECMAScript proposal for a pipe(line) operator syntax in JS, but it is only at stage 2. It is not quite the same as Haskell's Do Notation but is an alternative to some of the functionality we have implemented above.

Taking recursion to the next level

Tracy Gilmore — Sat, 05 Aug 2023 15:12:59 +0000

If you have read any articles on the topic of recursion the chances are they used either the Fibonacci Sequence or the calculation of Factorial numbers as the case study. Both of these are rather trivial and somewhat pointless examples outside of their academic purpose. However, to ease in this article we will briefly reprise the examples.

Simple examples

Factorial numbers

This must be the simplest example as it takes in, manipulates and returns a single value.

function factorial(n, fac = 1) {
    return n ? factorial(n - 1, fac * n) : fac;
}

OK, I was not quite truthful about only taking in a single value but the second value is not expected to be supplied on the initial call but provided as part of the recursive calculation, to pass the running total from one call to the next. But, why bother, the following iterative example works just as well.

function factorial(n) {
    let fac = 1;
    for (; n; n--) {
        fac *= n;
    }
    return fac;
}

Those less familiar with the concept of recursion might find the iterative approach easier to understand but that is the lesson. Familiarity with the concept and its benefits (and drawbacks) will help you understand how and when a recursive solution is beneficial. I would argue these simple examples don't go quite far enough.

Fibonacci sequence

This has to be The textbook example, yet I have included it because it is the next minor step in the learning process.

function fibonacci(n, curr = 1, prev = 0) {
    return n ? fibonacci(n - 1, curr + prev, curr) : curr;
}

As before, this function is only expected to be supplied with a single argument when called, the others are used to pass values in subsequent calls. And again, the iterative version is hardly complicated.

function fibonacci(n) {
    let curr = 1, prev = 0;
    for (; n; n--) {
        [curr, prev] = [curr + prev, curr];
    }
    return curr;
}

Neither of the examples given above demonstrate the power recursion has to offer.

Micro operations

A couple of features common to most recursive functions can be characterised as the micro and macro operations.

The micro operation (or operations) is at the core of the function and performs the primary function. In the Factorial function it is n => n * (n - 1), in the Fibonacci function it is:

add prev to curr
copy the previous curr to prev for the next cycle (which might require a temporary variable).

Macro operations

This is the mechanism by which the recursion is repeated and, more importantly, concluded. In all versions of the ‘simple examples’ we repeated the micro operation n times (while it is truthy), reducing it by one (decrementing) each cycle and terminating when n = 0, or in the context of JavaScript when n became falsy.

A more realistic example

I recall reading somewhere (citation required) that around 80% of source code is engaged in converting data from one format to another. I assume the remainder is concerned with interfacing and persistence. A common feature of the ‘simple examples’ is they have a single path (cycle) of execution but recursive functions can have more paths, as will be demonstrated in the following example.

NB: The following example has questionable utility but again serves very well as an academic exercise.

objectFlat()

Following in the mode of the Array.flat method, objectFlat is intended to reduce the level of property nesting of a given dataset.

The function makes the following technical assumptions:

The input data will be a valid JSON object. By that I mean it is an object that could have resulted from a JSON.parse operation and contains no cyclic references.
The data types will be limited to the primitive values: null, Boolean (true, false), numbers and strings.
The number of supported data structures is also limited to Arrays and Objects although these can be nested, their flattening being the purpose of the function.
Arrays will only be flattened when they are object properties, not nested in top-level arrays.
Flattened array items will be converted into object properties with the name in the following format propertyName[array Item Index].
Likewise, flattened object properties will be renamed objectPropertyName.subPropertyName.

Should you be interested, the source code contained within this article can be found in my GitHub repo.

Yet again, the function accepts a number of arguments but only the first parameter is expected on the initial call. Subsequent calls might provide the second to indicate the returned value is for an object property. More on this in a moment.

Primitive values

The initial argument can be an array, object or a primitive data value. In the case of the latter, the value is returned unchanged. So the following four examples will be output from the function without modification (just JSON stringified to aid presentation).

Test input:

const primitiveValues = [null, true, 42, 'epsilon'];

Test results (JSON stringified):

(index)	Values
0	'null'
1	'true'
2	'42'
3	'"epsilon"'

Arrays

Arrays are only flattened when they are assigned to the property of an object, otherwise the array remains unchanged in structure. If the items in the array are objects, they will be flattened as will be demonstrated later.

Test input:

const arrayTests = [
    [],
    [null, true, 42, 'epsilon'],
    [
        [null, true, 42, 'epsilon'],
        [null, true, 42, 'epsilon'],
        [null, true, 42, 'epsilon'],
    ],
];

Test results (JSON stringified):

(index)	Values
0	'[]'
1	'[null,true,42,"epsilon"]'
2	'[[null,true,42,"epsilon"],[null,true,42,"epsilon"],[null,true,42,"epsilon"]]'

Objects

Nested objects are flattened. By that I mean, the properties of the inner object are renamed and copied to the outer object.

Test input:

const objectTests = [
    {},
    {
        alpha: null,
        beta: true,
        gamma: 42,
        delta: 'epsilon',
    },
    {
        zeta: {
            alpha: null,
            beta: true,
        },
        eta: {
            gamma: 42,
            delta: 'epsilon',
        },
    },
    {
        theta: {
            zeta: {
                alpha: null,
                beta: true,
            },
            eta: {
                gamma: 42,
                delta: 'epsilon',
            },
        },
    },
];

Test results (JSON stringified):

(index)	Values
0	'{}'
1	'{"alpha":null,"beta":true,"gamma":42,"delta":"epsilon"}'
2	'{"zeta.alpha":null,"zeta.beta":true,"eta.gamma":42,"eta.delta":"epsilon"}'
3	'{"theta.zeta.alpha":null,"theta.zeta.beta":true,"theta.eta.gamma":42,"theta.eta.delta":"epsilon"}'

Notice how the name of the flattened object property is composed of the property path for the value but as a string. In the third example, the input property theta.eta.delta is converted to a single top-level property of the name “theta.eta.delta” in the output object.

But what happens when we combine arrays in an object or objects as elements of an array?

Test input:

const compositeTests = [
    {
        alpha: null,
        beta: true,
        gamma: 42,
        delta: 'epsilon',
        zeta: [null, true, 42, 'epsilon'],
    },
    [
        {
            alpha: null,
            beta: true,
            gamma: 42,
            delta: 'epsilon',
        },
        {
            alpha: null,
            beta: true,
            gamma: 42,
            delta: 'epsilon',
        },
    ],
];

Test results (JSON stringified):

(index)	Values
0	'{"alpha":null,"beta":true,"gamma":42,"delta":"epsilon","zeta[0]":null,"zeta[1]":true,"zeta[2]":42,"zeta[3]":"epsilon"}'
1	'[{"alpha":null,"beta":true,"gamma":42,"delta":"epsilon"},{"alpha":null,"beta":true,"gamma":42,"delta":"epsilon"}]'

The first test case is an object containing a property zeta that has an array value. In the output object the property zeta is replaced with a property for each item in its array, with the name of the property being a composite of the property name followed by the array item subscript as follows: zeta[0]. However, in the last test case the output remains unchanged because the array is not an object property and the objects in the array only have primitive values.

Implementation

Now for the “meat” of the article, how was the objectFlat function assembled, how does it employ recursion, what does this let us and what are some of the pit-falls of recursion?

Under the hood

The initial version of the function will present an interface (signature) with only one parameter; we will consider the second parameter later. Within the function we determine if the supplied argument is an Array, Object or a primitive value. We detect arrays using the idiomatic (built-in) isArray method of the Array prototype. There is no equivalent built-in function or method for detecting Objects and there are a mass of edge cases to trip up hand-created isObject functions, but here is what we will use.

function isObject(obj) {
    return obj === Object(obj);
}

Once we have eliminated Arrays and Objects, we only have to consider primitive values.

function objectFlat(srcObj) {
    if (Array.isArray(srcObj)) {
        return srcObj.map(item => objectFlat(item));
    }
    if (isObject(srcObj)) {
        return Object.entries(srcObj).reduce((tgtObj, [key, val]) => {
            tgtObj[key] = objectFlat(val);
            return tgtObj;
        }, {});
    }
    return srcObj;
}

Primitive values are returned immediately and unchanged. Array items and Object properties get pushed back through the objectFlat function, but also remain unchanged. Notice how we have two places where recursion happens, array items and object properties (key, value pairs). In this example the macro operation is ‘self limiting’ (as long as there are no circular references). There will be a finite number of array items and object properties. For input that is already flat the function of course achieves nothing, but that is in itself a result. However, as you might expect, the real work happens when the data is nested.

However, the above ‘initial’ function still performs some recursion, except for the primitive value test cases. The three array test cases “recurs” 0, 4 and 15 times, respectively. The first case is an empty array so no items are processed, but the second example is populated with four primitive items, so the function is recursively called for each, to one depth of stack. The third Array example is nested with another array, each with four primitive values. In this example, each sub-array results in a recursive call (3), each of which will call the function a further four times, once for each primitive item (3 x 4). Fifteen recursive calls in total.

The four object test cases perform similarly. The first is empty so there is no recursion. The second example is a flat object of four primitive values so there are four recursive calls. The third and forth examples recurs all primitive values, which results in 6 and 7 function calls to a depth of three and four levels, respectively.

So, how do we go about ‘flattening’ array items and object properties?

There are two ways to access object properties: dot-notation and bracket-notation.

const example = {
    alpha: {
        beta: true,
        gamma: 42,
        delta: [
            'epsilon',
            'zeta',
            'eta'
        ]
    }
};

In the above example we can access the gamma property as follows:

// dot notation
console.log(example.alpha.gamma); // 42

// bracket notation - literal
console.log(example['alpha']['gamma']); // 42

// bracket notation - indirect
const propName = 'gamma';
console.log(example.alpha[propName]); // 42

The property names (as strings) can include dot delimiters and square brackets, which enables us to convert nested properties and array subscripts like this.

console.log(example['alpha.gamma']); // 42

// bracket notation - literal
console.log(example['alpha.delta[1]']); // 'zeta'

However, array items can only be ‘flattened’ in this way when they are the value of an object property. Encoded properties by extending the property name is the reason why we need a second parameter, to pass the name of the property (propName) for the next call.

function objectFlat(srcObj, propName)

Converting primitives

    return propName ? { [propName]: srcObj } : srcObj;

Encoding primitive values is trivial because there is no further recursion. The function either returns the unchanged value or wraps it in an object when a property name is supplied, which can be subsumed into the object of the caller.

Converting (flattening) array items

    if (Array.isArray(srcObj)) {
        return propName
            ? srcObj.reduce(
                    (tgtObj, item, index) => ({
                        ...tgtObj,
                        [`${propName}[${index}]`]: objectFlat(item),
                    }),
                    {}
            )
            : srcObj.map(item => objectFlat(item));
    }

Likewise, when a property name is passed into a call with an array srcObj, the propertyName is encoded in conjunction with the index of each array element. The value of the array elements will themselves be encoded through a further call. The result of the call will be assigned to the value of a property in a new object, with the encoded name.

Converting (flattening) object properties

    if (isObject(srcObj)) {
        return Object.entries(srcObj).reduce((tgtObj, [key, val]) => {
            const tgtProp = propName ? `${propName}.${key}` : key;
            return { ...tgtObj, ...objectFlat(val, tgtProp) };
        }, {});
    }

When a propName is supplied we encode it into the name of each property to produce a new object that is returned. Once received from the calling location the original property, and its object value, are replaced with the properties of the object returned by the call.

In summary

When the objectFlat function is called with an object containing nested objects or arrays, the function is called again for each nested property/element, along with the name of the property of which it is the value. When nested, event primitive values are passed in a recursive call. In all cases the function returns an object that is expanded deconstructed to replace the nested property.

In total there are three places within the objectFlat function where it calls itself, twice within the array processing and a third in the object processing. However, it is only the call in the object processing that supplies a property name, which is how we flatten the object.

Proper Tail Calls and Tail Call Optimisation

One of the criticisms of recursive functions is that, if they are not written correctly, they can result in a considerable call stack. Each time a function is called, memory is allocated and memory locations (addresses) stored in a memory structure, known as a call stack, so they can be restored as each function completes.

If the function is excessively recursive or unbounded the calls can exhaust the stack resulting in what is known as a stack overflow.

Tail Call Optimisation (TCO) is a facility of the runtime environment that greatly reduces call stack proliferation by replacing the call with the result but in order to use this facility;

recursive functions need to be written in a way they can take advantage of TCO and
the runtime has to support the facility.

The JavaScript specification (ECMAScript ECMA-262) defines how this can be achieved but as yet (July 2023) very few JS runtimes/engines exist that have implemented the mechanism. See section 14.6 of ECMA-262 of 2015.

But even if/when the runtime supports TCO, the code has to be written in a particular way to take advantage of it. Functions need to employ Proper Tail Calls (PTC). This means function return must be either a value or a pure function call. The function cannot be dependent on anything in the function from where it is being called. This is why the examples in this post pass data from one call to the next via parameters, to ensure independence.

Additional material

Full Stack To Do list, a step-by-step tutorial

Tracy Gilmore — Fri, 14 Jul 2023 12:34:58 +0000

Introduction

The To Do list has become the "Hello, World!" example to compare and demonstrate new JS frontend frameworks. But this tutorial is different as it does not involve a frontend framework, instead it will use vanilla web technologies for the user interface (UI). This is done so we can focus on the stack as a whole and the interfaces between tiers, which I consider to be the most important element of a stack.

Another reason why the To Do application is a good example, is it includes the four basic data manipulation (or CRUD) operations common to many business systems. The CRUD acronym stands for:

Create: adding a new item to storage.
Read: retrieving one or more items from storage.
Update: changing the data of one or more stored items.
Delete: removing one or more items from storage.

In our example the Update and Delete operations will only be applied to one ToDo item at a time and the Read operation will be used to retrieve all the ToDo items in storage.

Another common acronym in such systems is REST or RESTful, which describes an approach used to communicate between the web browser and the server (or API). There is more to REST than we need to discuss in this post but it is worth knowing how the CRUD operations align to the REST strategy. REST uses HTTP (Hypertext Transfer Protocol) to send requests to the server and receive responses in the client/browser (typically in that order). There are a variety of request types used to instruct the server of what is required, known as the HTTP Verbs, or Methods.

This is how CRUD operations can be mapped to the HTTP Verbs:

Create -> POST
Read -> GET
Update -> PATCH (PUT can also be used but they mean slightly different things)
Delete -> DELETE, well that is obvious.

Here is a link to an MDN page if you would like to learn more about HTTP Verbs/Methods.

Developing inside-out

By defining and even implementing the interface between two subsystems early, a more firm foundation is established for the construction of both sides either-side of the interface. As will be demonstrated, in a three-tier architecture (frontend (FE), backend (BE) and database (DB)) there are two subsystem boundaries; FE/BE and BE/DB.

Our backend will be little more than a two-way translation layer between the database and the user interface (UI). Later in this post we will identify other responsibilities of a backend but our implementation will be kept simple to demonstrate the fundamental machinery and concepts. It is worth noting the backend comes in two parts, web server and application server. Both json-server and Express are able to facilitate these roles from the same URL. This is very useful for our tutorial because we do not have to configure the server to manage Cross-Origin Resource Sharing (CORS). It is quite typical for production systems to separate these server roles for all sorts of good reasons but for now it would just create an additional complication.

Source code (caveat emptor)

I have published the source code for this article in a GitHub repo, but I have to issue a word of caution. THIS IS NOT PRODUCTION CODE. For reasons discussed towards the end of this post, the code is for educational purposes only and should not be used in production.

A step-by-step tutorial

Our example application will be based on the ME*N stack where M is a MongoDB (but it could be MySQL, or any other) database. E is for Express, which is a backend framework that sits on top of Node.JS. There are others but Express is extremely common and will meet our needs well. N is for the JavaScript runtime Node.js but Deno and Bun are possible alternatives. Finally the * is for the frontend framework such as Angular (MEAN), React (MERN) or Vue (MEVN), to name a few. However, in our example we will not be using a framework (to keep it agnostic and besides it is not that complicated), instead we will be using the native web technologies (HTML, CSS and JS).

There are six steps to the development in this tutorial, each building on the previous:

Simple array-based interface module - to establish the requirement but with volatile (lost on page refresh) storage.
Revised web-storage based interface - to provide local data persistence (during the browser session but could be longer).
User Interface (UI) with an API module revised to interact with a backend based on json-server - to establish the final version of the FE/BE interface, but with a mock backend and storage.
UI connecting to a Node-Express backend with the same FE/BE interface module as previously and reusing the first interface module (array-based) as temporary backend storage. There is a slight alteration to make it work BE but it is minor.
Evolution of the previous step to again employ json-server but for BE data storage only. This will reuse the FE/BE interface module from step three, again with a slight alteration.
Finally we swap out json-server for a MongoDB database, provided by MongoDB Atlas, which will require a new interface module based on the json-server version.

In the first two steps we will provide a unit test using Jest and a simple integration test (harness) using an HTML file. You will observe the UI is primitive and uses the development tools (browser console) to present the test results. The UI developed for steps three onwards is a little better and enables user interaction to exercise the CRUD operations in a more realistic manner.

Incremental development

This tutorial revolves around the development and evolution of the interfaces between FE/BE and BE/DB (crud-interface.js). The interface employs the revealing module pattern (RMP) to expose the functions:

createToDo
readToDoList
updateToDo
deleteToDo

The RMP also enables provision of dependencies such as data sets, functions and other modules in a form of dependency injections; a mechanism we will use to mock external facilities.

Step One: The first interface module - keeping it simple

To commence the development, we will adopt a Test-Driven Development (TDD) approach by creating the unit test first using the Jest test framework. However, our FE/BE interface will use the ES6 module (ESM) syntax instead of the Common JS (CJS) syntax default to Node.JS. Therefore, we need (as of Node 19.7.0) we need to use a command-line switch to enable ESM as follows.

node --experimental-vm-modules node_modules/jest/bin/jest.js

The crud-interface unit test will first import the interface module and a dataset in the form of the following JSON document.

{
  "toDos": [
    {
      "text": "Task One",
      "done": true,
      "id": "1"
    },
    {
      "text": "Task Two",
      "done": false,
      "id": "2"
    }
  ]
}

The unit test will perform four positive test cases, one per exposed method, plus three negative test cases for when the list is empty and then no toDo item is found when updating or deleting. The initial implementation of the crud-interface module is based around a simple array, which can be supplied when the module is instantiated. This means the unit test can pass in the toDos property of the above test data when we execute the default method of the module. The array CRUD operations are simple but we cannot replace the array itself otherwise the unit test will lose reference, instead we have to replace its content.

In addition to the unit test we also have an integration/functional test capability in the form of an index.html test harness. Instead of using the Jest framework, this test is exercised in the web browser in much the same way the module would be expected to operate. Therefore, we will be using the json-server package as a proxy web server to make all the resources available to the browser “online”; we will be using a similar strategy for the first three steps.

Step Two: Minor enhancement to improve data persistence

In step two we revise the internal workings of the interface module to swap out the array with the windows session web storage. The external/public interface will remain the same so the unit test will largely remain the same. The two key differences are:

The interface module dependency will be the window.sessionStorage namespace.
The unit test will mock the dependency, which enables the test cases to confirm actions from both sides of the interface.

As before, the implementation of the module will use the ESM syntax and there will be an HTML-based test harness to provide greater context and exclude the jest framework from the test environment, but will again use json-server to deliver content to the browser.

Step Three: Adding a User Interface with a proxy backend

We continue with the two-tier test approach in step three, but this time we will use json-server more extensively. Json-server will provide both a web server to deliver the page and an application server for access to data storage (in the form of a json file). In order to execute both unit and integration tests we need to run the server using the ‘start’ script. Once the server has completed loading, the ‘test’ script will perform the unit test of the revised interface module. Once complete, the integration test can be performed by accessing the test harness from the browser by loading ‘http://localhost:3000’ as before.

The crucial difference with the step-three version of the interface module, is that it uses the fetch API to interact with the server. The module’s only dependency is the URL of the API, but it could be extended to include the fetch API. This would make it possible to swap out the actual fetch mechanism for a mock equivalent in the unit test and avoid the need for the actual server to be running. Instead, the start script will reset the source data and start the server. The integration test is also extended to provide a more interactive (and realistic) user interface.

The UI presents a list of stored tasks and an input field to type the name of a new task. As might be expected, the Add button creates a new task in storage, Done updates the associated task to mark it ready for deletion, and the Delete button removes the associated task from storage. When the input field is empty the Add button will be disabled.

This exact version of the module will be used for the FE/BE interface of the next three steps.

Step Four: Connecting the UI to a production-grade backend with simple storage

In this and the following steps we will be using an interface module in the backend as well as the frontend. Where possible the BE/DB interface will be heavily based on a previously developed FE/BE implementation with one minor difference. The FE/BE modules use the standardised ECMAScript Module (ESM) syntax, which requires some configuration of NodeJS to facilitate testing. The BE/DB modules will employ the Common JS (CSJ) module system that is native to the NodeJS environment.

In this step, and from here on, the middle tier (backend) will be implemented using the ExpressJS framework on top of NodeJS. The Express component exposes the RESTful that maps requests from the frontend to the BE/DB interface. Step four uses a BE/DB interface module based on the very first implementation (step one). The only difference between the two implementations is the change from the ESM syntax,

export default function (toDoList = []) {

and the CJS module syntax

module.exports = function (toDoList = []) {

both expect the toDoList array as a dependency.

Step Five: Adapting the json-server FE/BE interface module for BE/DB use

Similar to the previous step, the next step repurposes an earlier implementation of the FE/BD interface module (from step 3) for use as a BE/DB interface using the json-server as a proxy database.

Again the only real difference is the change in module syntax.

From export default function (endpointUrl = 'http://localhost:3000/toDos'),
to module.exports = function (endpointUrl = 'http://localhost:3300/toDos').

You may also notice the only other difference is the change in port number from 3000 to 3300. This reflects the fact we are communicating directly with the database (proxy) instead of the backend server.

Step Six: Replacing the proxy storage with a production-grade cloud server

Finally, we replace the BE/DB interface and proxy database with the actual things.

In our example we establish a connection with a MongoDB database hosted by their Atlas service. The connection needs to be secured so parameters have been stored in a .env file of the following structure.

USER_ID=userId
USER_PWD=userPassword
DB_NAME=databaseName
DB_COLLECTION=databaseCollection

Making a production-grade version of the example stack

As is this implementation of the ME*N stack is exceptionally vulnerable to attack. There is no protection from error or misuse through the FE, which would be greatly improved by using a FE framework such as React, Angular, Vue etc. The FE/BE interface is also wide open to malicious actors. This can be improved by employing HTTPS to encrypt the communication path and implead “man-in-the-middle” attack. JWT can also be used to establish user authentication. The stack can also be made more robust and maintainable through the use of two Express middleware packages (Helmet and Mongoose).

Helmet helps “sanitise” the input, which might not have come from the UI directly. Mongoose is what is known as an Object Document Modelling (ODM), which defines a structure (schema) for the stored data, making it easier to manage in Express. These additions have been omitted from our example stack purely to simplify the tutorial and focus on the fundamental tiers and interfaces.

Software development on a Chromebook

Tracy Gilmore — Sat, 24 Jun 2023 20:48:34 +0000

> Not only is it possible, with some reasonable compromises it is even pleasurable

What was I thinking?

Despite being, or maybe because I am, a full-time Software Engineer, I regularly have a number of personal development projects on the go. They are seldom very elaborate but can be anything from a single file investigation to a small trial application. I like to document my projects, sometimes in the repo’s ReadMe file, other times in a separate on-line blog (dev.to), and occasionally both. Primarily this is for my own benefit but should someone come across them and find them useful, I would like to think they have all the information they require.

Dell G7 17" Intel i7

At home I have a large personal laptop with a 17 inch screen and loads of power/storage but it is far too heavy, cumbersome and valuable to take anywhere, so I usually take something smaller. I have tried smaller mobile devices (tablets) in the past but found they were only good for reading posts, email and watching videos. Attempting any form of development, even with a good WiFi connection, was a futile exercise. There were many reasons for this but one of the biggest was I just could not get on with the soft keyboard that took up half of what was already a small screen. So recently I decided to purchase a cheap and light-weight laptop to use when travelling.

ASUS 14 CX1400CMA

Of course the range and type of software development achievable on a low power device would be limited but I was genuinely surprised at how capable my chosen device was. In fact, the setup I have been using to write some experimental code (in JS) is very similar to that I currently use as a professional frontend developer on a medium-size project.

Why a Chromebook

I have always been quite impressed by the power to cost ratio Chromebooks offer. I have revitalised old laptops in the past by installing ChromeOS. However, they often incurred a similar weight issue as my G7 and needed to be plugged in all the time as the battery was long dead and not worth replacing.

Another reason for choosing a Chromebook over a Windows laptop was they are so different. My expectations of the lower powered device would be tempered. This would hopefully limit what I would ask of the machine to that within its capability and not what I really need a larger machine to archive. In the past I also owned a Windows XP Acer netbook, which was very good but I found it very frustrating because the Intel Celeron processor was not up to my unreasonable expectations.

Another consideration of being constantly reminded the machine was not running Windows, I would refrain from doing anything sensitive on the machine like online banking. Should, on my travels, the device fall into the hands of someone unscrupulous, I will not be providing them with a front door into my personal life.

For a few years I have been aware of on-line development environments such as JSBin, JSFiddle and CodePen. They have spearheaded on-line development and more recently a new breed of on-line resources have become available including CodeSandbox, Stackblitz and Replit. You can even access your GitHub repos directly through an in-browser (web) version of MS Visual Studio Code by pressing the full-stop (try it in one of your own repos). Of course there are also cloud offerings from Amazon, Google, Microsoft, etc. but they require a little more configuration and setup than I was happy to incur. Finally, there are two relatively new offerings in this space in the form of GitPod and GitHub Codespaces. I have signed up but not yet explored what they have to offer.

My hardware

As mentioned earlier, at home I have a personal Intel i7 laptop with a 17” screen, 8GB of RAM and 2TB of storage, running Windows 11. At work I also use Linux via AWS on a similarly spec'ed machine. So this is what I have become accustomed to using for professional software development.

I wanted a machine that was considerably lighter than my personal machine so I was happy to consider a screen size between 13” and 15”. Also, I did not need as much storage or the power of an i7 processor. However, a small Windows laptop, from a manufacturer I recognised, started at around £400, which was about twice what I was willing to pay.

Thinking I may have been unrealistic in my requirements I started to think slightly out of the box and reconsidered what I would use the machine for. It was then I found some Chromebooks in the £200 range. Yes, I was being miserly because this machine was only ever going to be used when I travelled and so had a high risk of being lost or damaged in transit.

My Experience

I will document my experience in three stages; what I was expecting as a minimum, what I was hoping might be possible and finally, what actually was possible.

Spoiler-alert: It worked out much better than I had hoped.

My initial setup (out of the box)

Hardware: ASUS CX1400CMA

14” screen
Full HD (non-touch) screen
Physical keyboard (very important)
Intel Celeron N4020
4GB RAM
64GB eMMC

Not exactly a powerful piece of equipment but cheap, light, quick (as it happens) and with superb battery life (12hrs); ideal when travelling.

Minimal expectations

Whilst travelling I want to be able to:

Research places to visit and plan my journeys
Keep an online journal of my adventures
Read articles that have been accumulating on my reading list for weeks
Experiment with some minor coding projects and document my findings.

All things well within the capability of a Chromebook. I was sure as long as I had a reasonable internet connection I would be happy, and I am.

My stretch expectation

As a professional software engineer I work with code for a living, 5 days a week 8hrs a day. However, as a principal software engineer opportunities to engage in coding myself are becoming extremely rare. Instead I am engaged in many other activities involved in professional application development (planning, designing, code reviews, mentoring, etc.), all of which I enjoy (so I am not complaining). What this does mean, however, is I also feel the urge to code (personal projects) and investigate new techniques in my personal time. To be fair I have always done so but these days it seems more important than ever, even when I am travelling.

In my downtime I was really hoping to be able to ramp up my personal investigations using more advanced online resources like CodeSandbox or Stackblitz. Fully aware of the limitations imposed on me by the Chromebook, I considered my expectations to be somewhat ambitious. I was pleasantly mistaken.

CodeSandbox (CS) worked exactly as you might expect with an up to date browser such as Chrome 112+. At this stage in my investigation I was not intended to capture my findings in a post like this. I opened CS with the intention of developing some example code to demonstrate the topic of another post and started by creating a new vanilla JS sandbox. Right away my eye was drawn to some menu options I did not recall seeing before.

I suspect the GitHub option has been available for some time but I never really needed it. However, on this occasion I thought I would take a look and clicked the button to be shown the following dialog.

As I usually sign in to CS using GitHub credentials, it did not surprise me when I was not prompted for them again but if you try replicating this action, you might. I entered the name of a new repo and pressed the 'Create repository' button and there it was, a Git repo accessible from my CS environment. I soon realised that what this offered me over using VS Code in the browser was a runtime environment; very handy. But what if you already have a sandbox, well then that is when the next “new” option becomes interesting.

Now for the icing on the cake

Before we venture into using VS Code with CS, there is another important change we should discuss and that is how there are now two environments. Older sandboxes and those created using some of the basic templates are known as Browser Sandboxes and run entirely client-side (within the browser), which might be all you need.

Newer Cloud Sandboxes run remotely on CodeSandbox's microVM technology, which has a free tier for personal and small team (up to five contributors) developments.

Clicking the VS Code button presents the following dialog to take the Browser Sandbox and create a new Cloud Sandbox from it, leaving the original untouched.

Once the Cloud Sandbox is created it will appear in the dashboard with a label. Inside the sandbox you are shown the project in the web version of VS Code, almost the same as the environment you get for CS repos in the browser but richer. Curiously, there was another reference to VS Code in the top-right corner of the screen with a drop-down menu.

Selecting the “Open in VS Code Insiders” option presents the following dialog.

If VS Code is already installed on the local machine, pressing the “Launch in VS Code” might try to install the “insiders” edition before opening the application to provide you with remote access to the environment over a Secure Shell (SSH) connection. But, look at the lower half of the dialog and you see something I found really interesting.

The cherry on the top

My first thought on seeing the installation instructions was “but I cannot install VS Code on this tiny machine, it is not available for ChromeOS from Microsoft nor is it through the Google Play Store”. I was wrong again but there was a little more to it.

The link in step 1 presented me with a couple of Linux varieties of VS Code to download. I thought “what’s the harm” and downloaded the Debian package assuming the website would be aware I was using ChromeOS and would not have offered it if it were unsuitable. I downloaded the package and started installation, which is when it got very interesting. Before installing VS Code, ChromeOS enabled its “Linux development environment” feature, which is a little like WSL (Windows Subsystem for Linux), and prepared a virtual machine (VM) for me. It went on to install and configure VS Code on my VM and provided an icon in the ChromeOS launcher.

The link in the second step commenced installation of the “CodeSandbox Extension for VSCode”. Once complete I noticed CodeSandbox was not the only plug-in that had been installed but Remote Explorer (and associated SSH extensions) had also been installed.

What this means now is, I have VS Code (IDE) installed locally, a Cloud Development Environment (CDE) provided by CodeSandbox and a secure connection between them. I also have the option to protect projects in a GitHub repo and potentially develop an application with a small team.

As stated previously, as long as I have a reasonable internet connection I can develop code and execute it in an environment almost certainly more capable than my local machine. I would argue many developers these days would find their jobs almost impossible without ready access to the internet and it is only in extreme cases when such restrictions are imposed on a team.

Conclusion

So I have to admit, in retrospect, I think the title of this post should have been “Software development through (or “using”) a Chromebook”. A (justifiable) criticism hardware designers have of software engineers is that, as soon as they increase the computational capability of computers, software developers go and squander the benefit, but not in this case. The availability of Cloud Development Environments (CDE) greatly reduces the need for a powerful laptop in order to engage in quite significant software development.

Since starting this article I have discovered the same (or similar) hardware is available, at a similar price, but with Windows 11 installed. However, I do not regret getting the Chromebook. It is very quick to start and run and appears to be very battery efficient.

Supplemental

I have been using the hosting environment provided by Netlify for years. I am aware there are several available including Vercelli and Heroku, to name a few, but Netlify was the first I tried and it continues to work well. It links into my Git repos on BitBucket and also offers Functions-as-a-Service (FaaS) for experimentation and small projects. Netlify is also a prime advocate of the JAMStack architecture that is gaining considerable interest.

In a very recent development Oracle have made available “always free” online Linux servers. For more details I suggest you check-out this YouTube video and I will let Gary Explain as he does it so much better.

DEV Community: Tracy Gilmore

Using a WebWorker in the browser to create a plug-in architecture

Test scenario

Example Plugin: addition.js

List of plugins (manifest.json)

Exploring candidate solutions

Some less effective alternatives

DOM inject

Dynamic imports

Output

A worked example (Web Worker)

The Scenario

Simplified solution

Manifest

Application (Index.html script)

Plug-ins.js

Word of Caution

An NPM dependency check list

My NPM dependency checklist

Road to self-driving development

Six levels of Automation

Resources

Experiments with and elements

The aesthetic limitation

Just an idea: Using a prefix with id attributes for direct JS access

I’ve given up on Windows and you could too

Back story

So why (and how) did I change to Linux

The secondary push

I was convinced it was time to change

Closing thoughts

Using a tagged template to simplify Regular Expressions

Introduction

Template Literals to the rescue

Origins

Evolution

Utilising a Tagged Template

Hear the TL raw

Terminology

Helper functions

Some Use Case examples

Use Case scenario

Example One: hard coded

Example Two: interpolated

Example Three: restricted

Conclusion

Some guidance for using RegExp

Some additional advice when in JS:

Streaming SIMD in JS

Things to know before reading this post.

Quick recap of sim-simd

Enhanced SIMD (esimd)

Exceptions

Testing

The test instruction and data

Implementation

Supplementary code

executeInstruction

transform

cacheSurplus

permute

No-Cache mode

Cached mode

Matrix mode

Bringing it all together

Update April 1st, 2024 (no fooling)

Exploring DO Notation in JS

Introduction

The use case

Converting temperatures between C and F

Let’s convert Celsius to Fahrenheit, and back again

The Canonical imperative approach

Going functional

Generic and Specific Operations

Do Notation in JS

"Cracker" diagrams

Example 1: DOing it with specific functions

Example 2: DOing it with generic functions

Example 3: Using formatted (string) input

Example 4: Conditional processing of formatted input & output

Experiments with
and
elements

Example 6: `DO_WITH`