wj-config: Loading Configuration From Disk or Fetching From the Web

Originally published here.

Heredia, Costa Rica
2022-10-23

Series: One Configuration for all your JavaScript, Article 2

In this quick article I would like to present the various ways JSON data can be loaded from external sources, such as disks or from the web.

Preface

wj-config is a JavaScript configuration package written in TypeScript and transpiled to ES Modules inspired by the .Net Configuration system where any number of data sources can be specified in the order of preference, and the end result is a merged version of all configuration data sources.

The supported data sources are:

1. Environment.
2. Raw JSON strings.
3. POJO's (Plain Old JavaScript Object).
4. Dictionaries (flat objects that follow a naming pattern).
5. Single values.
6. Fetched JSON data.

If so desired, anyone can add more data sources as the library is extensible.

The library provides many features. Among those are:

1. An environment object that provides information about the current environment (development, staging, production, etc.) inspired by .Net's own IHostEnvironment interface.
2. Configuration value tracing (tells you which data source set the value found in the final configuration object).
3. URL functions that can automagically build URL's using the data found in the configuration object like host name, scheme, port and paths. The functions even replace route values and dinamically support query string addition, all while URL-encoding all values.

wj-config can be used in browser frameworks like ReactJS as well as NodeJS. One configuration library to rule them all.

Import the JSON File

As seen in the first article of the series, one simple way to read a configuration JSON file is to simply import it. This works in NodeJS as well as ReactJS and probably several other frameworks and libraries. My current experience is based on the two aforementioned ones so this is what you'll see here.

Importing in ReactJS

This is super simple, as shown in the first article of this series.

import mainConfig from './config.json';

That's it. Now your config file is parsed and its result stored in the variable named mainConfig.

Importing in NodeJS

NodeJS has this additional syntax requirement: An assert object that specifies the type of import.

import mainConfig from './config.json' assert { type: 'json' };

The end result is the exact same thing: mainConfig contains the contents of the JSON file.

This process, simple as it is, comes with drawbacks. The most obvious one, and one that you probably can notice in the first article of the series, is that it is statically loaded. You cannot opt-in, based on runtime code execution, whether or not you want or need the JSON data in said file. This is particularly important when you want to conditionally load configuration data. If you were to use this method to load environment-specific JSON's, you would have to load all your environment-specific JSON's, even though your system runs only under one of those environments.

Load a JSON File From Disk

This method is applicable to NodeJS only because browser applications do not have access to the end user's file system, and even if they did, what's the point? Your configuration file is not there. Your configuration file probably resides in the server, so you better fetch() it (see next section).

Anyway, back on topic. You can use NodeJS's integrated fs module to load a JSON file.

This example demonstrates how to read and parse a JSON file, and then add it as configuration source:

import fs from 'fs';
import wjConfig from 'wj-config';

function loadFile(fileName) {
    return new Promise((rslv, rjct) => {
        fs.readFile(fileName, 'utf8', (err, data) => {
            if (err) {
                rjct(err);
            }
            else {
                rslv(data);
            }
        });
    });
}

const fileContents = await loadFile('./package.json');
const contents = JSON.parse(fileContents);
const configPromise = wjConfig()
    .addObject(contents)
    .build();
const config = await configPromise;
console.log(config);

NOTE: The example is intentionally written like this and not as an importable module merely so you can paste this code as the main code of a test NodeJS project and run it right away. Usually we export config, then we import it where needed.

Now you can do things like an if statement to conditionally load a JSON configuration file and optionally include it, or build the JSON file's name in runtime. You can even make the presence of said file optional and simply supply an empty object if the file is not found, like in the following variant:

import fs from 'fs';
import wjConfig from 'wj-config';

function loadJsonFile(fileName, isRequired = false) {
    return new Promise((rslv, rjct) => {
        fs.readFile(fileName, 'utf8', (err, data) => {
            if (err && isRequired) {
                rjct(err);
            }
            else if (err) {
                rslv({}); // Empty object.
            }
            else {
                rslv(JSON.parse(data));
            }
        });
    });
}

const contents = await loadJsonFile('./package.json');
const configPromise = wjConfig()
    .addObject(contents)
    .build();
const config = await configPromise;
console.log(config);
console.log('End.');

NOTE: This variant also includes an extra change: Parsing is done inside the file-loading function, so the function is renamed to loadJsonFile.

Now if the file-loading code encounters an error (such as the file not existing), the returned promise still resolves successfully, only with an empty object.

This is my preferred method and how I recommend loading environment-specific data sources from disk (or fetched): If not found, return empty objects so as to not disturb wj-config's fluent syntax.

About Dockerized NodeJS Applications

Dockerized containers usually don't have anything else running in them, so the file system is expected to be super stable and predictable. This means that we can enhance the above function with fs.existsSync() to check if the file exists, and leaving the error for more serious matters only.

function loadJsonFile(fileName, isRequired = false) {
    return new Promise((rslv, rjct) => {
        const noFile = !fs.existsSync(fileName);
        if (noFile && isRequired) {
            rjct(new Error(`File ${fileName} does not exist and is required.`));
        }
        else if (noFile) {
            rslv({});
        }
        else {
            fs.readFile(fileName, 'utf8', (err, data) => {
                if (err) {
                    // You probably want to log the error or something.
                    console.log(err);
                    if (isRequired) {
                        rjct(err);
                    }
                    else {
                        rslv({});
                    }
                }
                else {
                    rslv(JSON.parse(data));
                }
            });
        }
    });
}

Fetch a JSON File

NodeJS has had an implementation of fetch() since some v17 release, and in v18, while still experimental is enabled by default and working fine in my limited testing.

Because of this, wj-config does come with built-in functionality to fetch JSON data and is very simple to use.

To add a data source object from fetched JSON data, you simply need something like this:

import wjConfig from 'wj-config';

// In ReactJS, you could put configuration files in the root of the /public folder.
const myConfigUrl = '/config.json';

const configPromise = wjConfig()
    .addFetchedConfig(myConfigUrl)
    .build();
const config = await configPromise;
console.log(config);

Just like that. I actually recommend that for ReactJS applications, you put the environment-specific configuration files in the /public folder, and maintain the main configuration file in the /src folder. If you do this, you import the main configuration and you fetch the environment specific configuration.

import wjConfig from 'wj-config';
import mainConfig from './config.json'

const myDevConfigUrl = '/config.dev.json';

const configPromise = wjConfig()
    .addFetchedConfig(myDevConfigUrl)
    .build();
const config = await configPromise;
console.log(config);

And that's about it for basic fetching, but let's explore a bit more some more advanced options available when fetching configuration data.

The Overloads of addFetchedConfig

There are 2 overloads of the addFetchedConfig() method:

addFetchedConfig(
    url: URL,
    required?: boolean,
    init?: RequestInit | undefined,
    procesFn?: ProcessFetchResponse | undefined
): IBuilder

addFetchedConfig(
    request: RequestInfo,
    required?: boolean,
    init?: RequestInit | undefined,
    procesFn?: ProcessFetchResponse | undefined
): IBuilder

Both overloads provide the same functionality and they differ in one thing only: The data type of the url parameter (the first one). They closely mimic fetch() because, after all, this is a wrapper of said function.

Optionally Fetching

Probably the most helpful of parameters, besides the obvious url one, is the required parameter. It is super simple to use: Pass a Boolean value that indicates whether or not your application can only function properly if the data is fetched. So basically pass true if you cannot live without this configuration data, or false if your application will be OK even if the fetching process fails.

If you passed true then wj-config will throw an error if the data cannot be fetched. That simple.

Passing Headers, Tokens and Whatnot

If your fetching of the configuration data requires special data like custom headers and such, you can make use of the init parameter. Refer to the MDN's documentation on fetch() to understand how it is used. addFetchedConfig() does nothing with this except passing it to fetch(). Hint: It is the options part in that MDN document.

Custom-Process the Fetched Data

Ok, so most of the time you won't need this at all. 99.9% (or more) of the times, you'll be obtaining JSON data directly from the URL you specify in the call to addFetchedConfig() and that's it. But what if the data comes in a different format? What if it comes in JSON, but you only need pieces of that JSON?

Well, then you must provide your on data-treatment function. This function of yours will receive the response object obtained by calling fetch() and it is up to you to do whatever you need to do to obtain the sought configuration data.

As an example, let's imagine that we want to fetch a large configuration document from a larger system, but we only need one or two sub-sections of that data. No problem. Do this:

import wjConfig from 'wj-config';

const enterpriseConfigUrl = 'https://my-enterprise.api.example.com/config/full'; // Or whatever.

const configPromise = wjConfig()
    .addFetchedConfig(enterpriseConfigUrl, true, undefined, async (response) => {
        const data = await response.json(); // Obtain the full JSON response.
        // Now return only the needed pieces.
        return {
            subSection: data.clients.myClientsConfig
        };
        // Or spread.  Whatever works for you:
        return {
            ...data.clients.myClientsConfig
        }
    });
    .build();
const config = await configPromise;
console.log(config);

By providing the custom processing function we are able to download a big configuration data file and just pick the piece or pieces that we want. In the example above we are taking just the configuration that we want for the specific client.

HINT: This function can also be used to transform the hierarchy of the data into a hierarchy that is more of your liking. After all, wj-config is about compiling configuration data in a hierarchical manner, and if you have a data source that does not conform to your preferred hierarchy, feel free to use this transformation function to create a version that does adhere to your likings/needs.

---

In the coming articles I'll cover the Environment class and how it can be used to stop hardcoding the environment-specific configuration file, like in the examples seen so far. With that knowlegde you will be able to start using wj-config everywhere.