Adventures of a Hobbyist ~ Part Three

Thinking about conf files

What is this?

This is the third part of my 'learning to code' series, it's been slow progress but all progress is progress. If you're interested in reading about the project here are the first 2 parts of the series:

• Preamble
• First week of Learning

What are conf files?

I have a background with Linux and server maintenance, it's quite common for programs to have .conf files to contain all of their settings. For web applications, I've made in the past, I've always hardcoded the database's location and then stored all other settings there. That approach doesn't really work when you want to make open source software as it needs to be easy for anyone to use it in their infrastructure.

To this end, I started thinking about using .conf with node. It should be easy enough, I thought, we have fs built in and that's all we need. So I set about writing a little something to handle reading the conf file and getting the settings imported. This is what I ended up with.

The conf file

"General": {
  "name": "ignis"
},
"MongoDB": {
  "host": "localhost",
  "port": "27017",
}

The function to read it

module.exports = {
  "loadConf": async () => {
    const fs = require('fs');
    const ConfFileLoc = "ignis.conf";

    async function getConfFile() {
      return new Promise((resolve) => {
        fs.readFile(ConfFileLoc, (err, data) => {
          if (err) throw err;
          resolve(JSON.parse(`{${data}}`));
        });
      });
    }

    let conf = await getConfFile();
    return conf;
  }
}

So what's the problem?

I tried to access this data from another file, as I imagine it would be helpful to be able to read the .conf file from anywhere, but it would only tell me there was a pending promise. I worked out a way around this but it felt a bit hacky and I'm sure there's a simpler solution to this.

const ch = require('./conf_helper');

(async () => {
  let conf = await ch.loadConf()
  console.log(conf)
})()

As you can see I've set the whole section as an async function but I'd have to have the entire file, apart from imports, in the async function which feels like a bad idea.

I want to help.

If you want to help me out you can either leave a comment on this post or respond to my GitHub issue about it. If I'm doing it a totally stupid way feel free to tell me, my aim is to learn how to do things properly.

Side note.

You may have noticed I mentioned MongoDB in my examples above I'm not certain I want to use this yet. Historically, I've always used MySQL but MongoDB was suggested as a better solution so I'm looking into it. If you have any input for database discussion the GitHub issue is the place to go.

Fin.

Thank you so much for reading this far and coming with me on my journey of learning. If there is anything I can do to make these posts more interesting/engaging please let me know in the comments, I really appreciate any, and all, input I get and want to make this a good series for you, the read, as well as myself.

Thanks again,
Andrew.

Comments

[Avalander]

You could use dotenv to load configuration variables easily from a .env file. It's very convenient for database urls and api keys and similar things.

If you want to keep your solution to read the config file, though, there are a couple of things you can do about the pending promise:

1. The easiest is to load the file with fs.readFileSync. If you are loading the file only once at start up, there is no reason why you need that call to be async.

2. If you really need a start up operation that is asynchronous (like connecting to a MongoDB instance), you can hook your start up function at the point where the promise resolves and inject the resolved value to all the code that is dependant on it.

const startApp = conf => {
    // Do everything that you need to start your app here.
}

const conf = await ch.loadConf()
startApp(conf)

I don't use async/await, but you can take a look at how I start an express server after initiating a connection to MongoDB here.

To reiterate though, I strongly suggest you use dotenv to store database urls and similar data that you don't want to commit to your version control system. And if that doesn't suit you for any reason, there is likely no good reason not to use fs.readFileSync to read your config file at the app start.

[Andrew Bone]

dotenv looks like it does exactly what I need, thank you so much 🙂

[Devin W. Leaman]

Personally for configurations, I usually have a config.json file with the values I need, since in JavaScript you can require the file and most intellisense modules will pick up all the keys in the file.

Another option is to set environment variables using something like PM2, which is what I use to actually run my node modules. You create a simple pm2.config.json and add the environment variables in there and on startup the values are available with process.env.VARIABLE_NAME.

If you're interested, I'll put a more detailed example in your git repo for you to take a look at.

[Devin W. Leaman]

Also, the beauty of the config.json is there's no more need for the fs module 😊

[Andrew Bone]

Any examples would be great, thank you 🙂

[Andrew Bone]

I think I found a solution that works for me, I like the idea of the file being .conf and I found this Node Module. It had a few bits I didn't like so I wrote a wrapper around it. Here's what I've ended up with.

class ConfHelper {
  constructor() {
    const Conf = require('conf');
    const EM = require('events');
    this.events = new EM.EventEmitter();
    this.config = new Conf({
      configName: 'ignis',
      fileExtension: 'conf',
      cwd: '.'
    });
    this._getConf();
  }
  _getConf(key, type) {
    this.conf = this.config.get();
    this.events.emit('change', key, type);
  }
  createKey(key, val) {
    if (this.config.has(key)) throw `${key} already exists, please use updateConf`
    let keyVal = this.conf;
    let layers = key.split('.');
    let name = layers[layers.length - 1];
    for (let i = 0; i < layers.length - 1; i++) {
      if (!keyVal[layers[i]]) keyVal[layers[i]] = {};
      keyVal = keyVal[layers[i]];
    }
    keyVal[name] = val;
    this.config.set(layers[0], this.conf[layers[0]]);
    this._getConf(key, "create");
  }
  deleteKey(key) {
    if (!this.config.has(key)) return
    this.config.delete(key);
    this._getConf(key, "delete");
  }
  updateKey(key, val) {
    if (!this.config.has(key)) throw `${key} does not exists please use createConf`
    if (this.config.get(key) === val) return
    this.config.set(key, val);
    this._getConf(key, "update");
  }
}

module.exports = ConfHelper;

And here is the test I wrote to go with it.

const ConfHelper = require('./conf_import');
const ch = new ConfHelper()

ch.events.on('change', (key, type) => {
  let event =`
  -------
  type    ${type}               
  key     ${key}                
  newVal  ${ch.config.get(key)} 
  -------`;
  console.log(event)
});

ch.createKey('General.version', "v0.0.1");
ch.updateKey('General.version', "v0.0.2");
ch.deleteKey('General.version');

Which outputs

  -------
  type    create
  key     General.version
  newVal  v0.0.1
  -------

  -------
  type    update
  key     General.version
  newVal  v0.0.2
  -------

  -------
  type    delete
  key     General.version
  newVal  undefined
  -------

Thanks for all your help 🙂

[Devin W. Leaman]

In my experience it's a million times easier to stand up a free tier MongoDB through their atlas service. If you need a local instance, I believe two or three commands is all it takes and you have it up and running locally and only accepting connections from the local host.

[Andrew Bone]

Their main reason was that they prefer the syntax but my own research suggested that it was faster and easier to find online hosting for.