Vector's VRL introduction (chapter 2)

In a previous chapter we learned how to run VRL and made our first "Hello World" application.

It was mostly trivial, except for the unusual use of a "dot" (e.g., .message).

The Dot

The VRL program is a bit different from usual programming languages. The main convention is that there is a . variable, which contains the event (for which the VRL application is running). The value of . at the end of the program execution is the return value.

. is not just a variable name. As you've noticed, you can access it the way you do in most other programming languages (object.property), but when you access the content of the dot, you don't need to write ..message; you write just .message. By the way, in VRL, if you have other variables, you can do variable_name.property too.

Let's look at our previous program, which was cleaning stuff from the dot.

.message = "Hello World"
del(.timestamp)
del(.host)
del(.source_type)

We are modifying the event. But we can do better.

. = {"message": "Hello World"}

Or like this:

. = {}
.message = "Hello World"

In those two examples we are not modifying .; we are replacing it with new content consisting of the object (also called a dictionary, if you are coming from a Python background). We either construct it in place (JSON notation) or create an empty object and set (create) a key message with the value "Hello World".

Assignment and Reassignment

Reassignment rules allow using . in expressions that replace . (or any other variable).

. = {"message": "Hello World", "old_message": .message}

If we run it with input test, we get the expected result:

{
  "message": "Hello World",
  "old_message": "test"
}

We also can create new variables with a simple =.

old_dot = .
. = {}
.message = "Hello World"
.old_message = old_dot.message

Each assignment is a deep copy (that means, it copies all data recursively).

temp = .
. = {}
.message = "Hello World"
.old_dot = temp
temp.message = "replaced"

With the same input (test) we get this output. Note, there is no "replaced" in the output, because we copied temp into .old_dot before changing temp.message.

{
  "message": "Hello World",
  "old_dot": {
    "message": "test"
  }
}

Some ergonomic hints

You can try those examples from the console, with the command line:

vector --config example.yaml  -q|jq .

(put code examples into example.yaml.)

You also can try the VRL playground: https://playground.vrl.dev/

Please note that my initial example contains this source:

sources:
  stdin:
    type: "stdin"

And it internally creates an object with a few fields: host, message, test, source_type, stdin, timestamp.

If you decide to run code from the playground, you need to create an event (not just the word test), so the minimal input for the playground is {"message": "test"}.

There can be a bit of confusion between not-a-JSON input and the event object at the VRL start, but it's easy to explain and understand:

Everything Vector passes between sources, transforms, and sinks is always an object. Sources can take non-structured input (like stdin or exec), but they wrap unstructured text into an object. They also append metadata; that's how you get timestamp and host.

Second: VRL allows you to put spaces and newlines in most places (but not all!).

These are two equivalent programs:

. = {
        "message": "Hello World",
        "old_dot": .
      }

and

. = {"message": "Hello World", "old_dot": .}

But this is NOT a proper VRL program:

.
  = 
    {}

You also can have a trailing comma in objects:

. = {
        "message": "Hello World",
        "old_dot": .,
      }

This may be silly at first glance, but it makes git diff much cleaner if you decide to add a line after the last one:

. = {
        "message": "Hello World",
        "old_dot": .,
        "something": "else",
      }

My advice is to use a trailing comma because it reduces syntax errors when you move lines around or add/remove them.

In the next chapter we will discuss types, fallibility, and progressive typing.