Painless YAML Templating

❤️ YAML ?

YAML is ubiquitous. Wether you love it or hate it, you have no choice but using it. It established itself as the configuration format for all things cloud/devops/serverless (choose your buzzword).

The main characteristic of YAML is the role played by semantic indentation. It makes lazy humans save typing braces to delimit block they would indent anyway.

YAML	JSON
hello: foo: bar	{ "hello": { "foo": "bar" } }

Humans like YAML because it focuses on the content

Complexity Increase

Automation is at the heart of everything in the raise of Infrastructure-as-Code, Cloud, DevOps, GitOps, etc.
Containerization also contributes to the exponentially growing number of configuration combinations.

Inevitably, a need for structure and factorization arised.

Text Templating to the Rescue

That's why most of the tools ingesting YAML configuration support some kind of templating.

The first generation approach relies on existing text templating libraries (Jinja, Go templates, Helm, ...).
It is a clever but lazy move because it does not work well when indentation comes into play.

Whoever struggled with toYaml in Helm charts is a victim of YAML text templating.

From a user experience point of view, the opening and closing symbols that YAML avoided come back in the templating language.

  {{ if eq .Values.favorite.drink "coffee" }}
  mug: true
  {{ end }}

Alternative: a Dedicated Language

To avoid the drawbacks of text templating, one can adopt a configuration language providing the programmatic features needed to tackle the complexity and generate the YAML. This is typically the approach taken by Jsonnet. It is a powerful language that natively understands the data structure.

  Martini: {
    ingredients: [
      {
        kind: $['Tom Collins'].ingredients[0].kind,
        qty: 2,
      },
      { kind: 'Dry White Vermouth', qty: 1 },
    ]
  }

It can output in many formats, including YAML.

So, everyone should adopt this no? Why is it not the case?

Try to switch to a new language, you will just add a new one to your already bloated Babel tower.

It is difficult to adopt Jsonnet because it cannot be introduced progressively. You have to make a dramatic switch in order to get the benefits. Few organizations are capable of driving such changes in a top-down way.

As all the examples and litterature about YAML-based tools are written in YAML, using Jsonnet imposes to systematically translate them.

And note that these unwanted opening and closing braces are back again...

Enters Structural YAML Templating

So, the truth is in a middle way.

Could we write plain YAML and add programmatic features to it in some undisturbing way?

YTT does this with a language written in YAML comments.

  #@ for/end echo in data.values.echos:
  - name: #@ name(echo)
    image: hashicorp/http-echo
    args:
    - #@ "-listen=:" + str(echo.port)
    - #@ "-text=" + echo.text

The structure of YAML is preserved and understood by the tool.
Is this the ultimate definitive way to solve the problem?

As a developer, don't you a feel slight discomfort when you write code inside comments?

As YTT is designed as replacement of text templating for configuration files, it provides a set of operations optimized for the task. Some features like overlays are powerful, some others like a single source for template values are limitating.

Leveraging YAML Tags

Thinking about this, I reminded some work I did with YAML about ten years ago. It is a dependency injection system for Python à la Spring Framework. It uses the YAML tag system to create a configuration corresponding to Spring's Application Context. This YAML, thanks to the tag system and anchors is directly deserialized as wired singletons forming the application structure.

YAML's strength is in the tags

With YAML tags in and mind inspiration from YTT and Jsonnet, came the design of Yglu ᕄ !?.

Input

greeting: !- Hello
greeter: !()
  length: !? len($)
  message: !? $_.greeting + ', ' + $  
names: !-
  - world
  - foo
  - bar
messages:
  - !for $_.names: !()
     - !? ($_.greeter)($)          

Output

messages:
- length: 5
  message: Hello, world        
- length: 3
  message: Hello, foo
- length: 3
  message: Hello, bar

Find more examples in the online playground and the test samples.

With the tag system and a powerful expression language (YAQL, in this case), YAML structural templating features can be truly idiomatic.

Wrap-Up

Leveraging YAML tags for creating a balanced DSL with a functional expression language brings YAML templating to a next level.

It allows for introducing YAML factorization progressively, integrate with existing YAML tooling and provide an idiomatic user experience.

It is probably the correct base for the tools to come. Maybe with other expression languages, but certainly with YAML tags.

Yglu ᕄ !?

Comments

[Paweł Kowalski]

Most definietly one of the ugliest syntaxes ive ever seen (and ive used AWS nightmare with Attr:Join etc.).

I would go huge lenghts to not use it, including string concatenation in ejs or mustache ;)

PS. Since yaml is compatible with json, its very easy to just write your configs in json (or function that spits out json) and then at the end of it, just convert it to yaml, if the tool cannot understand json.