DEV Community: Kirk Lewis

Template File Processing in Go

Kirk Lewis — Thu, 29 Aug 2019 10:35:03 +0000

In a previous post I demonstrated how to get a Go template's interpolated string result. However, if you just want to interpolate and then write the string result straight to file, this post demonstrates how.

The Template File

The following text can be saved to a file templates/greeting.tmpl.

{{.Greeting}} {{.Name}}!

The template string above contains two annotations .Greeting and .Name which is enough for this demonstration.

Execute and Write

The first argument of the text/template package's Execute function uses an io.Writer interface. As such, if we pass this argument an os.File, its write function will be used to write the processed template bytes straight to file.

Add the following code to a file named process-file.go .

package main

import (
    "os"
    "text/template"
)

func main() {
    // variables
    vars := make(map[string]interface{})
    vars["Greeting"] = "Hello"
    vars["Name"] = "Dev"

    // parse the template
    tmpl, _ := template.ParseFiles("templates/greeting.tmpl")

    // create a new file
    file, _ := os.Create("greeting.txt")
    defer file.Close()

    // apply the template to the vars map and write the result to file.
    tmpl.Execute(file, vars)
}

For brevity, I have omitted error checking in the code above by using an underscore

Run the file

go run process-file.go

Running the line above should create a file named greeting.txt. Check the contents of this file.

cat greeting.txt

Hello Dev!

Thank you for reading! The code used in this post can be found as a Github gist here.

Metrics with Prometheus StatsD Exporter and Grafana

Kirk Lewis — Wed, 31 Jul 2019 15:46:04 +0000

This post explains how to convert StatsD metrics using the Prometheus StatsD-exporter, and later visualising them with Grafana.

I explain from the perspective of an application which is instrumented with StatsD but is monitored by a Prometheus server. As such, I will use the terminal as a pretend application emitting metrics ; Prometheus will monitor the application and collect any metrics ; and finally Grafana will be used to draw a graph by using Prometheus as a data source.

Overview of Technologies Used

This section briefly covers why each technology is being used but to learn more, you can read about each using the links in the Recommended Reading section.

Metrics

During a system's runtime, measurements taken of any of its properties are known as metrics. These metrics are typically in the form of data points associated with some information to give it context. Metrics are things like the number memory bytes used, number of website visitors, the time a task took to complete, etc.

StatsD metrics allow such information to be recorded and represented using a text format specification. Below are two StatsD metric lines each representing how long an API request has taken.

# metric format: <metric name>:<value>|ms
test_api_call.orders-api.timer./v1/orders:84|ms
test_api_call.orders-api.timer./v1/orders:95|ms

In the above example, two API calls to GET /api/orders are measured at 84 and 95 milliseconds each. The |ms is used with StatsD timer metrics. StatsD also supports other metric types like Counters, Gauges, Histograms and more.

Why dot-delimited?

To record information about the metric, in this case the name of the API and its endpoint, the <metric name> part is dot-delimited so those details can be captured.

Later you will see how the Statsd-exporter will help us capture the dot-delimited parts we are concerned with.

Prometheus for Monitoring

Prometheus can be used to monitor the performance of an instrumented system such as, a microservice, a host server, etc. The system will need to expose its metrics in order for Prometheus to collect them - this is also known scraping. The scraped metrics can then be visualised or queried using the Prometheus expression-browser. Prometheus also has an AlertManager which can send alerts about certain metrics being monitored - it has support for Slack, PagerDuty, email and more.

The following diagram shows Prometheus scraping an application's metrics which are exposed on the/metrics path. Grafana can then use these metrics for visualisations.

Statsd-exporter

The statsd-exporter is used to convert Statsd metrics into Prometheus metrics. This is because the StatsD text-format represents metrics differently to Prometheus' exposition format which is represented like this:

<time series name>{<label name>=<label value>,...}

The following diagram shows how a StatsD metric, is converted to Prometheus metrics by using an exporter's mapping rules.

It is very common for a system to record its metrics in a format different to Prometheus, therefore there are many Exporters written which allow such metrics to be converted to Prometheus' time series notation

Where did count, sum and quantile come from?

The StatsD-exporter automatically converts timer metrics into a Prometheus summary. The summary contains extra information like the count, sum and each quantile for the current observation. By default the quantiles are 0.5, 0.9 and 0.99.

In the image above, 10 API requests took 80 milliseconds each to respond. As such a summary is generated with these details:

_count of 10.
_sum of 0.799999999999999 for that observation.
and each _quantile where 0.5 (50th percentile) is the median.

Now there is an understanding of what each technology does in relation to each other, we can get started implementing our test application.

Configure Prometheus and Statsd-exporter

In this section the configuration files for both Prometheus and the statsd-exporter are created.

Create the Directory and Files

Create a directory named .prometheus to contain the files prometheus.yml and test-mapping.yml as follows:

# I have created mine in $HOME/.prometheus/
mkdir ~/.prometheus && cd $_
touch ./{prometheus,test-mapping}.yml

Run tree or find . to verify the files.

Prometheus Configuration

This gives us some basic configuration for the Prometheus server such as static service discovery, and how often to scrape each discovered service.

Add the following to the prometheus.yml file and save it.

global:
  scrape_interval:      15s
  evaluation_interval:  15s

scrape_configs:
  # optional: this makes the metrics available to us about Promethus itself.
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  # tells Prometheus to scrape metrics an address over port 9123
  - job_name: 'test_metrics'
    static_configs:
      - targets: ['host.docker.internal:9123'] # see statsd-exporter further down
        labels: {'host': 'test-app-001'} # optional: just a way to identify the system exposing metrics

Prometheus also supports dynamic service discovery which is useful if you deploy applications on AWS ECS or other scalable cloud solutions.

Statsd-Exporter Configuration

This is the file used by the Prometheus statsd-exporter so it knows how to convert StatsD metrics into Prometheus metrics.

Add the following to the test-mapping.yml file and save it.

mappings:
    # usage: test_api_call.product-api.timer./v1/product
  - match: "test_api_call.*.timer.*"
    name: "test_api_call"
    labels:
        api_name: "$1"
        api_endpoint: "$2"

Startup the Statsd Exporter

Before visualising any metrics with Grafana, we can check whether our Statsd metrics are being converted correctly.

To start, run the the official Prometheus statsd-exporter Docker image, and then send it metrics over port 8125.

Running

Make sure you are inside the .prometheus directory.

docker run --name=prom-statsd-exporter \
    -p 9123:9102 \
    -p 8125:8125/udp \
    -v $PWD/test-mapping.yml:/tmp/test-mapping.yml \
    prom/statsd-exporter \
        --statsd.mapping-config=/tmp/test-mapping.yml \
        --statsd.listen-udp=:8125 \
        --web.listen-address=:9102

What this does

listens for metrics sent to an address on port 8125 - e.g. echo "hello:1|c" | nc -w 0 -u 127.0.0.1 8125
converts Statsd metrics to Prometheus ones by using the test-mapping.yml file created earlier.
expose the metrics for Prometheus to scrape - in this case over host port 9123 - container port 9102.

After running the image you should see output like this:

Startup Prometheus

Now the Prometheus server can be started so it can scrape the metrics exposed on port 9123.

Recall that the prom-statsd-exporter container, is exposing metrics on container port 9102 which were remapped to host port 9123.

To start, run the official Prometheus Docker image.

Running

Make sure you are inside the .prometheus directory.

docker run --name=prometheus \
    -p 9090:9090 \
    -v $PWD/prometheus.yml:/prometheus.yml \
    prom/prometheus \
        --config.file=/prometheus.yml \
        --log.level=debug \
        --web.listen-address=:9090 \
        --web.page-title='Prometheus - Test Metrics Demo'

What this does

runs Prometheus server and tells it to scrape metrics from the targets defined in prometheus.yml
allow us to visualise the metrics using the Prometheus browser and even query them using PromQL
allows Grafana to use Prometheus as a data source over address port 9090

After running the image you should see output like this:

The above informs us that the prometheus.yml file was loaded, and Prometheus knows about the two targets to scrape.

Verify the Targets are Up and Running

Go to localhost:9090/targets. Each one endpoint for each target should be up.

We can start sending metrics and prometheus will scrape them.

Sending some Metrics

Now try sending a couple metric over localhost IP address and port 127.0.0.1:8125 using the lines below.

echo "test_api_call.orders-api.timer./v1/orders:89|ms" | nc -w 0 -u 127.0.0.1 8125
echo "test_api_call.orders-api.timer./v1/orders:68|ms" | nc -w 0 -u 127.0.0.1 8125

Now goto http://localhost:9123 to view the metrics. If you have updated your hosts file to resolve host.docker.internal to localhost then you can visit http://host.docker.internal:9123 instead.

Search for test_api_call and you should see a Prometheus summary similar to the following.

Visualise the API Call Metrics in Grafana

This final section looks at starting up Grafana on port 3000, adding a Prometheus data source, and creating a Graph to visualise the API call metrics.

Start by running the official Grafana Docker image.

docker run -d --name=grafana -p 3000:3000 grafana/grafana

The default username and password for Grafana are both admin and admin respectively.

Adding a Prometheus Data Source to Grafana

To add a Prometheus data source and add a Graph you can follow the instructions at the Prometheus website - instead of me repeating the same instructions here.

Plotting the average request duration
Within the the section "Creating a Prometheus Graph" where it mentions 'Entering a Prometheus expression' into the Query field, use the expression rate(test_api_call_sum[1m]) / rate(test_api_call_count[1m]) as seen below. This expression calculates the average request duration for the last minute using the Prometheus summary for test_api_call

Optionally, you can also update the legend field to {{api_name}} - {{api_endpoint}} as seen below also - doing so will display the legend like this: orders-api - /v1/orders.

Emit Metrics Continuously

Now to save us having to send metrics manually - just to update the graph more frequently, we can write a simple command-line statement which will send a metric every second. This will be done in two shells, in order to simulate API calls to our fictional orders-api and invoice-api.

Below the shuf command is used to generate a random range of milliseconds between 50 and 150.

Shell 1: orders-api

while true; do echo -n "test_api_call.orders-api.timer./v1/orders:$(shuf -i 50-150 -n 1)|ms" | \
    nc -w 1 -u 127.0.0.1 8125; done

Shell 2: invoice-api

while true; do echo -n "test_api_call.invoice-api.timer./v1/invoice:$(shuf -i 50-150 -n 1)|ms" | \
    nc -w 1 -u 127.0.0.1 8125; done

use ctrl+c to stop either loop

The Visualisation

After a few seconds you should see the graph drawing. And some time later - in my case 5 minutes your graph should look similar to the image below. Make sure the Dashboard's refresh rate is at least 5 seconds.

How do I emit metrics from a real application?

There are StatsD client libraries written for Go, Node, Python, Perl and more. See link below:
https://github.com/statsd/statsd/wiki#client-implementations
Some of the libraries also default the host localhost and port to 8125 so you can get up and running quickly.

Node StatsD Client Example

This example uses the NPM module node-statsd.

const StatsD = require('node-statsd');
const statsdClient = new StatsD(); // defaults to localhost:8125

statsdClient.timing('example.response_time', 80); // record 80ms response time
statsdClient.increment('example.site_visit');     // record a new site visitor count +1

Can I just emit metrics to Prometheus format instead of StatsD?

Yes, it is possible instrument an application to record metrics in Prometheus format. This approach eliminates the need to use a Prometheus exporter sidecar and the extra work of creating a mapping file.

To instrument your application with Prometheus, use one of the official or unofficial client libraries.

Conclusion

Getting Statsd metrics out of an application (terminal for this demo) and into Grafana, only required the following:

two configuration files.
running statsd-exporter, Prometheus and Grafana.
emitting metrics over a host address and port with UDP using the Statsd line protocol.
setting Prometheus as a data source in Grafana and configuring a graph.

That was it! The majority of this post is mostly explaining some essentials which I think are very important to understand. Although the terminal is used as a pretend application 'to emit metrics', the principles are the same for a real application. So whether you write your application in Go, Node, Python, etc, you can just use a StatsD client library to emit metrics from it.

Go Text Template Processing

Kirk Lewis — Sat, 27 Apr 2019 23:37:09 +0000

I recently worked on a project where I used Go's template package for parsing and interpolating templates. After processing the templates, I needed to use the interpolated string result for saving new file content, and also naming directories and files. This post is a walkthrough of the functions I wrote to help process text templates, then afterwards each function is demonstrated.

Go Project Setup

Since I am demonstrating these functions outside the project I created them for, I've created a simpler project for this post as follows:

go-template-processing/
├── main.go
├── src
│   └── demo
│       └── template
│           └── process.go
└── templates
    └── got.tmpl

Create the Template File

Add the following text to the file templates/got.tmpl. This file template will be processed later.

{{.Name}} of house {{.House}}

{{if .Traits -}}
Traits:
    {{- range .Traits}}
    * {{. -}}
    {{end}}
{{end}}

Anatomy of the Template

The annotations like {{.Name}} and {{.House}} will be replaced with their respective values, during template execution. The other annotations like {{if}} and {{range}} know as Actions, will also be evaluated. The hyphens - to either side of the {{ and }} delimiters, are used to trim any space before and after them respectively. The dot . represents the current element in the range of Traits being iterated.

Set the GOPATH

cd go-template-processing
export GOPATH=$PWD

Interpolating the Parsed Template

To separate the concern of first parsing then interpolating a template file or string, I've created a function in the file process.go which does the interpolation first.

Go refers to the this interpolation as "executing the template", where the template is applied to a data structure.

package template

import (
    "bytes"
    "text/template"
)

// process applies the data structure 'vars' onto an already
// parsed template 't', and returns the resulting string.
func process(t *template.Template, vars interface{}) string {
    var tmplBytes bytes.Buffer

    err := t.Execute(&tmplBytes, vars)
    if err != nil {
        panic(err)
    }
    return tmplBytes.String()
}

The process function should be easy to reason with given its comment, annotations and implementation.

The argument vars is typed to an empty interface, so that any type like map or Struct with fields corresponding to the template's annotations can be used. Any {{.Annotation}} without a corresponding field is replaced with <no value>, or panics if it fails evaluation.

Writing the process function above is a pre-emptive extract method refactoring done to avoid DRY occurring later in the code.

Processing a Template String

The next function added to the process.go file, will process a template string by creating a new Template and then parsing the given string str argument. The new Template assigned to tmpl is then interpolated using the private process function created earlier.

func ProcessString(str string, vars interface{}) string {
    tmpl, err := template.New("tmpl").Parse(str)

    if err != nil {
        panic(err)
    }
    return process(tmpl, vars)
}

This function will be demonstrated later.

Processing a Template File

This function is very similar to ProcessString but it processes the file indicated by the fileName argument.

The ParseFiles function can accept more than one file name argument. However, I only need to parse one file at a time, so I am using a single fileName argument.

func ProcessFile(fileName string, vars interface{}) string {
    tmpl, err := template.ParseFiles(fileName)

    if err != nil {
        panic(err)
    }
    return process(tmpl, vars)
}

That's it, both functions can now be consumed by client code.

Using the Process Functions

The ./main.go file below demonstrates how both process functions are used.

package main

import (
    "fmt"
    "demo/template"
)

func main() {
    // data structure the template will be applied to
    vars := make(map[string]interface{})
    vars["Name"] = "Brienne"
    vars["House"] = "Tarth"
    vars["Traits"] = []string{"Brave", "Loyal"}

    // process a template string
    resultA := template.ProcessString("{{.Name}} of house {{.House}}", vars)

    // process a template file
    resultB := template.ProcessFile("templates/got.tmpl", vars)

    fmt.Println(resultA, "\n")
    fmt.Println(resultB)
}

Running go run main.go should output the following:

Brienne of house Tarth

Brienne of house Tarth

Traits:
    * Brave
    * Loyal

Both the template string and file have been successfully processed. The resulting string can be now used for an email body, the contents of a file, etc.

Supplying Vars via a Struct

Once again, since vars is typed to an empty interface, a struct can be used here also.

I've added the following snippet just below the fmt.Println(resultB) statement in main.go.

// using a Struct
type Westerosi struct {
    Name   string
    House  string
    Traits []string
}

jorah := Westerosi{"Ser Jorah", "Mormont", []string{"Brave", "Protective"}}
resultC := template.ProcessFile("templates/got.tmpl", jorah)

fmt.Println(resultC)

Running go run main.go again should now output the following:

Brienne of house Tarth

Brienne of house Tarth

Traits:
    * Brave
    * Loyal

Ser Jorah of house Mormont

Traits:
    * Brave
    * Protective

Conclusion

Working with Go templates is pretty straight forward out of the box, making it easy to build your own project specific functions. While I have only covered a specific task in this post, you can read more about the Go template package at https://golang.org/pkg/text/template.

The code used in the article is available on Github.
Go version used at time of writing go version go1.12.1

Thank you for reading.

String Interpolation of Constants in Perl 5

Kirk Lewis — Mon, 01 Oct 2018 21:06:57 +0000

This post demonstrates how to use Perl 5 constants with String Interpolation and Regular Expressions. I will briefly explain how String Interpolation works in Perl 5, especially with constants. Finally I explain how to use constants with Regular Expressions, and a use case for this.

String Interpolation

In Perl 5, any scalar or list variables embedded within double-quotes will be substituted with their values at run-time. This type of string is known as an Interpolated String, and it gets resolved in a process known as String Interpolation.

Example:

my $PI   = 3.14;
my @nums = (3, 2, 1);

say "scalar: $PI - list: @nums";

The above outputs:

scalar: 3.14 - list: 3 2 1

Alternatively, Interpolated Strings in Perl 5 can contain an identifier wrapped in either ${} or @{}, and depending on their type, they get resolved accordingly - in this case scalar and list respectively

Example:

say "scalar: ${ PI } - list: @{ nums }";

outputs:

scalar: 3.14 - list: 3 2 1

In the above string, Perl looks-up and finds each identifier i.e. PI and nums within the current namespace, and they are resolved.

Since the expressions ${} and @{} highlight the intent of Interpolation more clearly, I will use them both going forward.

However, a Constant created with the Perl 5 constant prama is Sigilless, as such inserting one into a double-quoted string, will not be resolved. See below:

use constant DEV_OPS => 'dev_ops';
my $status = 'online';

say "Team ${ DEV_OPS } is ${ status }";

The above results in the following output:

Global symbol "$DEV_OPS" requires an explicit package name...

The above occurred because unlike $status, the constant DEV_OPS does not have a sigil, so Perl cannot look it up and resolve it - with regards to quotes ""

Resolving the Constant

To make the Interpolated String aware of any constants to be processed, a backslash \ is used to reference it as follows:

say "Team ${ \DEV_OPS } is ${ status }";
# or if we use more than one constant
say "Team ${ \DEV_OPS } - ${ \DEV_OPS } is ${ status }";

Outputs:

Team dev_ops is online
Team dev_ops - dev_ops is online

List Context

Constant values are evaluated in List context so [] can be used to reference them also. Then use the appropriate interpolation expression as follows:

say "Team @{ [DEV_OPS] }...";

However, I find it simpler to use backslash \.

Using Constants in Regular Expressions

Lets say you have three constants, and each represents a different colour as follows:

use constant BLUE  => 'blue';
use constant GREEN => 'green';
use constant RED   => 'red';

Now, if you have a list of colours @colors which you want to validate against the three colours listed above, you may write something like this:

my  @colors = qw(blue yellow orange red);
my  $colors_regexp = qr/^(blues|green|red)$/; # typo intended...

say $_ =~ $colors_regexp
    ? "$_ is valid"
    : "$_ is invalid" foreach @colors;

The above outputs:

blue is invalid
yellow is invalid
orange is invalid
red is valid

Now blue in @colors is actually valid, but the typo blues in the Regular Expression, causes the match to fail. Whilst this typo is intentional, it could also happen by mistake, and be missed altogether.

Solution

Since each colour is defined as a Constant, it is safer to compose the Regular Expression using each one. This is done the same way I explained earlier, by using a backslash \ to reference each constant.

# lets make the regular expression a constant also
use constant COLORS_REGEXP => qr/^(
    ${\BLUE}  |
    ${\GREEN} |
    ${\RED}
)$/x;

To make the Regular Expression easier to read, I've used the x option. This option allows the Regular Expression to be written on new lines, and white spaces are ignored also.

Finally lets replace the Regular Expression $colors_regexp with the COLORS_REGEXP constant.

my  @colors = qw(blue yellow orange red);

say $_ =~ COLORS_REGEXP
    ? "$_ is valid"
    : "$_ is invalid" foreach @colors;

The above outputs:

blue is valid
yellow is invalid
orange is invalid
red is valid

Conclusion

String Interpolation using Constants in Perl 5 works the same as for scalars and lists, with the exception of using a backslash \ to reference the them. In Perl 6 however, you can just write say "${ constant_name }" and that's it! Perl also has the sprintf function, which allows formatting of variables embedded within strings.

Thank you for reading, and the code used in this post is available on Github.