SlimIO Architecture #2 - Configuration

Hey!

Let's dive into a new article of the SlimIO Architecture series! Today we're going to talk about the configuration files of our Agent and Addons.

We have started the SlimIO project with the creation of a package to manage configurations because this is one of the most important and core component of the product.

We absolutely wanted from the very beginning:

• A reactive configuration with realtime hotreload capacity.
• A safe configuration for our runtime and for integrators/administrators.
• Being able to migrate with no human interaction.
• Support both JSON and TOML.
• Bring on the table a first-class tool for addons developers.

It's a package specially thought and designed for a software oriented need.

Reactivity and hot-reload

Being able to reload a configuration on demand (or even automatically) without having to reboot the entire addon (or the agent) is an essential feature nowadays for monitoring products.

It's also a way to take into account the impact of a configuration change within the product from the beginning which allows developers to build their code without to have to worries about implementing their own abstractions.

Some products just push away this subject because there is to many side effects with the file system (they do not want to take any responsibilities). I guess this is one of the benefits of being at a higher level that those who have to deal with those issues in C or Rust (i mean without having to work directly with libuv or tokio for Rust).

---

The following example demonstrates how to stream modification for a given configuration path:

config.toml

log_level = 5

script.js

const cfg = new Config("./config.toml", {
  autoReload: true
});
await cfg.read();

cfg.observableOf("log_level").subscribe(console.log);

Right now each key of the configuration can be observed by the developer (it will return an Observable.).

⚠️ These API are still experimental and may evolve in the future!

By default the auto reloads is disabled because we know that this is a feature that is not always appreciated (depending on the severity of the monitored system).

The reload delay is also configurable with the Config constructor option payload (by default the delay is 500ms.).

Safe configuration

Being able to hot reloading is good but it can also lead to catastrophic situations where a configuration is corrupted or broken (because of human or system errors).

That's why we use JSON Schema to validate the payload of the configuration when we are reading and writing the configuration... SlimIO ensures that the configuration is always valid and up to date even in the worst situation.

For example a JSON Schema to validate the config.toml with the log_level key at the root.

config.schema.json

{
    "$schema": "http://json-schema.org/draft-07/schema#",
    "additionalProperties": false,
    "properties": {
        "log_level": {
            "type": "number",
            "description": "The level of the Addon logger",
            "default": 1
        }
    }
}

And this is IT! The @slimio/config package will automatically load the schema.

The schema allows a lot of new opportunities in the product:

• Dynamically generate the configuration UI for the ihm.
• Auto-complete and bring intellisense for administrators and integrators (work like a charm in VSCode).
• A self-documentation of each available key for a given configuration (not like products which bring magic keys out of nowhere).
• Allow to generate or edit the configuration from the SlimIO CLI (check this project).

And probably even more so in the near future.

Migration

Being able to migrate a configuration is also allowed by the presence of a schema. This allows developers to update their addons by following SemVer rules.

The migration will be managed by the Prism addon and will use the SlimIO Config-Migration package. Note that Prism will be capable to rollback any migration or addon upgrade (however we have not yet clearly defined the rules of this famous safeguard.).

The migration mechanism is a way to simplify patch and minor updates for developers.

TOML

At SlimIO we think it's time to improve the administration of configurations. We sincerely believe that YAML is a painful format and that JSON is very far from ideal for the job.

That why we are choosing TOML to be our white knight!

Example from the TOML github README:

title = "TOML Example"

[owner]
name = "Tom Preston-Werner"
dob = 1979-05-27T07:32:00-08:00 # First class dates

[database]
server = "192.168.1.1"
ports = [ 8001, 8001, 8002 ]
connection_max = 5000
enabled = true

[servers]

  # Indentation (tabs and/or spaces) is allowed but not required
  [servers.alpha]
  ip = "10.0.0.1"
  dc = "eqdc10"

  [servers.beta]
  ip = "10.0.0.2"
  dc = "eqdc10"

[clients]
data = [ ["gamma", "delta"], [1, 2] ]

# Line breaks are OK when inside arrays
hosts = [
  "alpha",
  "omega"
]

TOML is a good human readable format and eliminate most deep structure which are big pain for those who administrate all the product configuration files.

Bring tools to developers

Bringing the right tools to SlimIO and addons developers is for us an essential mission. We don't want the developers to have to re-invent the wheel on their side...

It's also a way to centralize contributions at one point to improve everyone's experience over the long term.

Conclusion

Our way of imagining the use of configurations is surely still far from perfect. In fact, nowadays many products have a single large configuration file (what isn't especially our philosophy).

From the beginning we have always had a vision of configuration less product (by the way a SlimIO agent can run with ZERO config.). Finally, our vision of configurations is very similar to our previous article -> Monolithic to Modular.

Moreover, our ihm will be specially built to allow you to configure your addons through an interface specially designed to guide you. The less time you spend in a configuration file, the more I think we have achieved our goal.

Best Regards,
Thomas