Extending YAML p.1 - structuring a meta-programming language

I've been working on Funcml for a few weeks now, a simpler way of applying mutations to data structures. For context, Funcml is part of a drive to replace Helm, the standard tool for managing Kubernetes manifests and releases.

A few thoughts

When it comes to Kubernetes manifests or declarative structures, structure languages like YAML or JSON are kings. While JSON is great for getting two machines to communicate, YAML is often preferred for being read and written by humans.

Structure languages like JSON or YAML are very practical for describing states (Kubernetes, Ansible are great examples...) but are fixed. No mathematical or generative functions are expected, as this is not their purpose.

This pushes us to use external tools as part of a DRY approach: Helm for Kubernetes is a good example, but all programming languages are generally capable of converting internal data structures into YAML or JSON. This simplifies the work, but every project is different and needs to implement a certain subset of generation functions - which is costly in terms of time and engineering.

Current solutions

Technologies have emerged to overcome this cost problem: jsonnet, cdk8s, Tanka, Pulumi... Based on languages familiar to developers, they provide libraries for mapping internal data structures to contextual objects such as Kubernetes manifests.

The complexity then shifts from the controlled generation of YAML files to design and development in a real programming language, with its semantic rules, best practices, testing phases and general complexity. Is this really a good idea?
On large projects, there's no doubt that it helps to consolidate the code, as long as you have competent people who have mastered these tools. But for all those intermediate cases where a simple YAML is not enough, and the integration of a complex YAML generation tool like Tanka is too costly, what's the alternative?

Reduce learning time

Here's Funcml: a tool that takes YAML or JSON files, identifies literally described functions and returns the result. It comes in the form of a command line and a library written in Ruby. The user writes a YAML or JSON file and uses value keys specially recognized by Funcml. Funcml then executes the associated functions and "mutates" the original data structures.

sum_values:
  _sum:
    - 1
    - 2
    - 3

By consuming the classic data structures of YAML or JSON, Funcml simplifies and accelerates their writing. It's easier for someone new to Funcml to generate their first files than someone new to cdk8s.

Almost a complete language ?

At the time of writing, Funcml's core library supports variable calls, mathematical functions, list transformations, dictionary transformations, string transformations and conditional rendering... But also string encoding, JSON serialization, file import, cryptographic functions and many other functions.

An approach close to LISP languages

Each structural block is called recursively by a mutate function. This allows the creation of a very composite language, similar to LISP languages: the first value returned will be the furthest one.

items:
  first: 1
  second: 2

value:
 _sum:
  - $items.first
  - $items.second
  - _add:
    - 1
    - 2
key:
  value: $value

# key: {value: 6}

in this example, the first function will be the final call to $value. $value refers to the value dictionary, which will thus start a _sum function. At this point, it retrieves the values ​​defined in the items dictionary and calls the _add function on 1 and 2 which then returns 3. The _sum is thus calculated and returns 6: value is thus equal to 6.

What are the advantages of using this structure?

Stupidly enough, there is no need for a special extension for its code editor: all Funcml is based on a correct YAML or JSON syntax; Funcml will refuse to work if the syntax is not respected!

Also, and unlike Helm for example, the final data structure must be correct to be rendered in JSON or YAML: this avoids development or debugging phases that end with 60 pull requests to correct an absurd syntax error.

Finally, Funcml wants to solve the complexity of generating YAML or JSON files that can be encountered in other solutions, as explained at the beginning.

Where is the project today?

The project is divided into three parts:

• funcml-core, which is a fully functional Ruby library that implements all the functions applied to data structures. Funcml-core can be integrated into any Ruby project ;)

• funcml-cli, which is a command line that allows you to use the Funcml-core library in a generic way, for example:
  

funcml-cli render ./file.yaml --mutations ./mutations.json --output-directory ./output

funcml-cli is not yet stable, only a few functions and arguments are functional today. The next few weeks will be dedicated to it in order to make it a truly functional tool.

• karist, which is a command line dedicated to Kubernetes - the alternative to Helm, remember? - and which is also in its early stages of development.

I don't want to rush the development of these tools. Generating and mutating data structures by recursion is not a very easy subject and design issues apply, such as overflow management, imposing mutation limits, circular calls...

Nevertheless, and if you are a Ruby developer and the project interests you, nothing prevents you from helping the projects ;)