DEV Community: Gabriel Lima

You get what you Measure: Understanding your applications health with Grafana, Loki and Prometheus

Gabriel Lima — Wed, 24 Apr 2024 14:55:00 +0000

Applications will fail, this is a reality from the universe. But these fails are not always catastrophic, sometimes they are just silent: a system that starts to answer incorrectly but just for some users, maybe responses start to get slow for an specific region.

Everyone that worked deploying systems enough have a story about how things "suddenly" went wrong on an Friday night after 10pm (a nightmare that haunts every DevOps dreams).

One way to avoid these kinds of problems (and others too) is to establish a good monitoring environment to track your application's health and availability. The objective of this tutorial is to help you create and understand this environment with some of the most famous tools for the task: Grafana, Loki and Prometheus.

Requirements

This tutorial uses Kubernetes to deploy the applications, however most of the content is focusing on the tools and related concepts rather than on deployment. So this should be good for everyone.

If you want hands-on practice you should have a running Kubernetes cluster (I used MicroK8s for this tutorial) and Helm (see how to install on Installing Helm tutorial). It is important that you understand the basics of these tools to fully understand.

Note that this is not an exhaustive list of features from the tools. See the sources section for a deeper understanding.

Capturing your metrics with Prometheus

One way to understand the health of our applications is to understand the metrics they produce. For example, if your database is slow you may want to know if it is not reaching the boundaries of your CPU/RAM by monitoring your CPU/RAM usage metric. If your web server is slow you may want to know if you are not receiving more requests that your system can handle. When you have one or two applications this can be easily done by getting these metrics manually for each tool. However when you have dozens or even hundreds of applications things start to get very difficult to manage.

How it works

Prometheus solves this problem by providing a centralized way to store and get these metrics from the application. For it to work each application should expose a port (by default 9090) that will contain its generated metrics in a specific human-readable format following the Prometheus Data Model. What Prometheus do is to use crawlers that will regularly get this information from the applications and send to the Prometheus instance. This is call a pull architecture.

Push Gateway

You can also work with a push architecture in Prometheus by using the Push Gateway. As the name says it works by the your application pushing its metrics to Prometheus (you can read more about how it works on "Pushing metrics").

This is not a thing to use always, in fact, the official documentation recommends to use that only in limited cases that you can understand more reading "When to use the Pushgateway".

Exporters

As I said before, Prometheus requires that the data is available for the crawlers in an specific format. But as you may be expecting not all tools follow theses rules. To solve this problem Prometheus have exporters.

Basically What an exporter do is implement a way to convert metrics from third-party systems to the Prometheus data format. One famous example is the JMX exporter can convert JVM-based application metrics. Another one is the Kubernetes Node Exporter that exports information about your Linux Kubernetes Nodes to the tool.

Querying the metrics

But with the metrics being on Prometheus, how can we actually see and use them? The answer is on PromQL, a query language that enables us to select and aggregate these metrics data.

In fact (and a little bit of a spoiler), this is what Grafana uses to query metrics from Prometheus and create the dashboards.

Alerting

Now, we just need to look for every log on the system and see if there is a problem, right? No, there is no need for that since Prometheus has also a built-in alert manager that will alert everything you want based on some metrics thresholds. For example, you can set an alert that triggers every time the response time is greater than 200ms.

Note that Grafana has a similar feature, so it is a good practice to just use one of these tools for alerts.

Configuring Prometheus

Prometheus can be deployed using the Prometheus Helm Chart. This helm chart contains a lot of features such as the already mentioned Push Gateway, Alert Manager and so on. For simplicity reasons of this tutorial I will not show all the Helm chart configuration but you can see a real example used by me here.

Crawlers and Scraping

As briefly explained before, Prometheus use crawlers to get information from the applications that generate these metrics. On Kubernetes these is done by specifying jobs on the scrape_configs section. You can see an example below:

serverFiles:
    ...
    prometheus.yml:
        rule_files:
            - /etc/config/recording_rules.yml
            - /etc/config/alerting_rules.yml
    scrape_configs:
        - job_name: prometheus
            static_configs:
                - targets:
                    - localhost:9090
        - job_name: kube-state-metrics
          scheme: http
          honor_labels: true
          static_configs:
                - targets:
                    - prometheus-kube-state-metrics.monitoring.svc.cluster.local:8080
        - job_name: kubernetes-apiservers
            kubernetes_sd_configs:
                - role: endpoints
            scheme: https
            tls_config:
                ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
                insecure_skip_verify: true
            bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
            relabel_configs:
                - source_labels: [
                        __meta_kubernetes_namespace,
                        __meta_kubernetes_service_name,
                        __meta_kubernetes_endpoint_port_name,
                    ]
                action: keep
                regex: default;kubernetes;https
            ...

On the example you can see that the first 2 jobs specify an static target (static_configs.target) that the crawlers should look for metrics. The first one is to scrape the Prometheus application itself on localhost and the second will look for metrics in the prometheus-kube-state-metrics.monitoring.svc.cluster.local:8080 address.

However the kubernetes-apiservers job acts differently. It uses a service-dicovery mechanism that will automatically find the metrics endpoints that respect some rules defined by the mechanism. Here we are using the kubernetes_sd_configs but others such as azure_sd_configs or http_sd_configs can be used. You can see more of this configuration on the scrape_config documentation.

If you go to your option status > targets inside you Prometheus instance you can see all the application (targets) that are being crawled and their status. It should look similar to this:

Capturing your logs with Grafana Loki (and Promtail)

But is not only about metrics that the SRE lives on. Another part of understanding how our applications are behaving is to read the logs that they generate, this is why you need Loki.

Loki works similar to Prometheus by having agents that scrape logs from the application (like Prometheus Crawlers), add metadata (labels) and streams them to a Loki instance. You can now read these logs using some tool, for example Grafana or the Loki Cli.

But differently from Prometheus, Loki does not scrape information automatically, so you need a way of sending you need a way of sending these logs to it. One of the most famous agents is Promtail. In a Kubernetes environment what it does is that it deploys a Daemon on every deployed node in the cluster. This Daemon reads the local logs generated by the pods and send them to a Loki instance (for those unfamiliar with Kubernetes, in a simplified way, this is similar to create a machine that will be deployed alongside your application machine and will be used to read all the logs).

There are a lot of agents that you can use such as FluentBit, Logstash and many others.

Configuring Loki

For Loki you can use the official Loki helm chart by Grafana. Again, you can see a real example by looking at this values.yaml file. Loki can be deployed in single binary where all the write, read and backend will be done in the same machine (monolithic). You can specify the values on write.replicas, read.replicas and backend.replicas to deploy in distributed mode. You can also store the logs in a file format or using a s3-like storage. In the example I use a MinIO deployed with other helm chart, but this helm chart provides a way to deploy a MinIO instance.

Configuring Promtail

To deploy Promtail, again you can use this values.yaml file as a base.

Promtail works with the concept of pipelines that will transform a log line, its labels and its timestamps in a way that is better for Loki. There are a lot of built-in stages such as
regex, json and many others. You can see more about pipelines and stages here and here.

Visualize all with Grafana

Finally, to make these metrics and logs actionable we need to have a way to visualize, aggregate and correlate them in a way that is easy for humans to understand. We also need to be alerted in case things go wrong. Grafana is a visualization and analytics software that can help us create visualizations (and alerts) from time-series databases. On our example these databases will be Prometheus and Loki, but many others could be used.

Deploy Grafana

Grafana is deployed by using the official Grafana Helm Chart. You can find more information about the chart on Deploy Grafana on Kubernetes. Again you can see a real example here.

Data Sources

What a Data Source do is specify where Grafana should get the data from, the example below shows that we are getting data from Prometheus and Loki. There are a lot of supported Data Sources that you can find here. Configuring a data source can be done directly via the UI or via the helm chart:

datasources:
  datasources.yaml:
    apiVersion: 1
    datasources:
      - name: Prometheus
        type: prometheus
        url: http://prometheus-server.monitoring.svc.cluster.local
      - name: Loki
        type: loki
        url: http://loki-gateway.monitoring.svc.cluster.local

Dashboards

We also want to visualize the information in ways that make it actionable, this is done by creating dashboards to show graphs and metrics for your applications. You can create your own dashboards via the UI or json files, however there are also a lot of community dashboards that you can install directly into Grafana (more information here).

dashboardProviders:
  dashboardproviders.yaml:
    apiVersion: 1
    providers:
      - name: default
        orgId: 1
        type: file
        disableDeletion: false
        editable: true
        options:
          path: /var/lib/grafana/dashboards/default
dashboards:
  default:
    # Ref: https://grafana.com/grafana/dashboards/13332-kube-state-metrics-v2/
    kube-state-metrics-v2:
      gnetId: 13332
      revision: 12
      datasource: Prometheus
    # Ref: https://grafana.com/grafana/dashboards/12660-kubernetes-persistent-volumes/
    kubernetes-persistent-volumes:
      gnetId: 12660
      revision: 1
      datasource: Prometheus

Dashboards are configured in YAML by setting a dashboard provider (read more on Dashboards) and the dashboards that you want to load, this can be a local dashboard configuration loaded as a json file or as the example shows above a dashboard found on the dashboards repository by selecting the id and the revision. For example, the first dashboard on the example loads the 12 revision of kube-state-metrics-v2 (id 13332) using the
Prometheus datasource. This should result in a dashboard that looks like this:

Sources

Setup your Python Environment with Pyenv and Poetry

Gabriel Lima — Wed, 01 Nov 2023 14:30:00 +0000

Dependency management in Python is very painful. Everyone that uses Python in multiple projects someday had a problem with package versions across the machine and even between team members. To solve these problems a lot of solutions appeared: venv, conda environments, Pipenv and many other tools that aim to create an isolated environment for you projects.

The objective of this quick tutorial is to help you easily setup a reproducible and isolated virtual environment. using Pyenv to manage your python versions and Poetry to manage your packages.

Managing your Python Versions: PyEnv

The objective of PyEnv is to help you install and switch between multiple Python versions in your system without worrying about breaking things.

Installing PyEnv

To install PyEnv you can follow their Installation tutorial. Personally I like to use their official automatic installer found here. For any problem I also recommend reading the Common build problems specially the prerequisites section.

Installing the Python Version

With PyEnv installed you can run the command below to obtain the list of possible Python versions that you can install:

pyenv install --list

The command above shows a huge list of python versions, you can choose whichever you need. For example, you can install Python 3.11.3 by running:

pyenv install 3.11.3

To know if everything went well you can check all the installed versions running:

pyenv versions

You should see an output similar to this:

system
* 3.11.3 (set by /usr/local/pyenv/version)

This means that are 2 python versions installed into your machine. The system is the one that comes with your OS. The other is the versions that we installed.

Choosing the versions

The * symbol means that this is the default version used whenever you call python (i.e. the global version). You can change the global version by running the command pyenv global. For example, to set 3.11.3 as the global version you can run:

pyenv global 3.11.3

You can also set the Python version in a folder, this is useful when you need a different versions of python for many projects. To do this use the pyenv local command. Following our example:

pyenv local 3.11.3

Managing your packages: Poetry

With Python installed in the correct version now you need a way to easily manage your packages in an isolated environment. Poetry is a tool to help manage your packages and its dependencies. It provides packaging and dependency resolution features (we are going to focus on this one) for you not to worry more about conflicts in your dependencies.

Installing Poetry

First you need to install the tool by following the installation guide.

Setup your project

Now you need to go to your project folder and run the poetry init command. This command will prompt you for some information including name of the project, authors, packages that you want to install from the beginning and other information. Note that this command will use the Python version setup by PyEnv giving preference to the local version.

poetry init

You will notice that the command will create 2 new files. The pyproject.toml is a human-friendly file introduced on PEP518 and expanded in some other PEPs (PEP 517, PEP 621 and PEP 660). This file store many information about your project that can be used by many tools of the Python ecosystem.

The poetry.lock files contains the installed packages (including their dependencies) and whatever is necessary for them to work. You should not edit this file manually.

If your project already have a pyproject.toml inside with [tool.poetry] sections it means that someone already setup the tool. You can skip the poetry init and go directly to run poetry install to install all the required packages in your machine.

Adding Packages

To add packages to the environment you just need to run the poetry add command. The command accepts a flag --group that is a way to organize the dependencies into groups. A common example is if you want to differ between production packages and development ones (e.g. test packages, code style and so on). In this case you can create a group called dev for the non-production environment. See an example below:

poetry add pandas==2.1.1             # Add pandas 2.1.1 on prod environment
poetry add pandas==2.1.1 --group dev # Add pandas 2.1.1 on dev environment

Using your environment

With your environment configured, you need to activate it to use every time you call python. This is as simple as running:

poetry shell

If you don't want to activate the shell you can also run any python command using the environment with poetry run inside the project folder. For example, to run an example python script using the environment you can do:

poetry run python some-script.py

For more Poetry commands and usage, see Poetry Docs

Final Thoughts

Since I started my python journey I had used many tools to manage packages: venv, Pipenv, requirements, conda environments. This setup was the one that I had less headaches with the many projects and versions that I worked on. This tutorial should help you achieve that.

Comment below if you use any other setup, if this tutorial helped you or any other comment that you want. Thanks for reading!

Sources

Creating an isolated and reproducible development environment: A tutorial on DevContainers

Gabriel Lima — Wed, 18 Oct 2023 13:35:07 +0000

You started on a new project, went through all the onboarding process, met your team, discovered the details of your new challenge and is now able to start the coding. Sounds exciting right?

But you know that before you start to define any variable in your IDE you need to setup all the tools, languages and environments for the project to work. And now the first challenge of you project is not to develop a new useful feature for the client, but to debug the tools that are not working as expected in your machine because of some computer magic.

This all sounds familiar. Right?

What are a Development Containers (or DevContainers)?

Development Containers are a specification that allows you to use containers as a full-featured development environment (read more on containers.dev). This enables users and teams to create a consistent, shareable, reproducible and isolated development environment.

Some benefits of using DevContainers are:

A faster onboarding for anyone that wants to join your project. Now a single command is sufficient to get people ready to contribute;
A shareable and reproducible environment that reduces problems related to different versions of packages and tools between team members;
An isolated environment from your OS that prevents your local OS from interfering with your project.

What you are going to learn here?

This tutorial will help you set a DevContainer for VSCode using the official extension by Microsoft. At the end you will be able to create your own custom container, use it on VSCode and personalize it without interfering in your team's environment.

Requirements

DevContainers work on Windows and Linux (including WSL2). For this tutorial to work you will need:

VSCode: You can install following this link;
Docker Engine: You can install following the steps on Docker Engine Install.

Setup your DevContainer

Step 01: Setup your .devcontainer folder

For VSCode to autodetect the container you should have at least a Dockerfile with your container's specification and a devcontainer.json file containing metadata and settings for the environment. All these files should be in a .devcontainer folder on the root of your project.

├── .devcontainer
│   ├── ...                 <- Any other related file that you want
│   ├── Dockerfile          <- Container Dockerfile definition
│   └── devcontainer.json   <- DevContainer's metadata and settings
└── ...

Step 02: Creating the Dockerfile

Here you can create a common Dockerfile with any base image, ARGs, ENVs and commands that you want your container to do. The example below will create a Ubuntu container with a python environment using Pyenv and Poetry. We also use the root user but if you want to use a non-root user you can follow this link.

FROM ubuntu:22.04
ARG PYENV_VERSION=2.3.18
ARG POETRY_VERSION=1.5.1
ARG PYTHON_VERSION=3.11.3

ENV DEBIAN_FRONTEND=noninteractive

# Copy custom scripts and set the required permissions.
COPY custom-scripts/ /tmp/scripts/
RUN chmod +x /tmp/scripts/*

# Basic packages and requirements
RUN apt update && apt install -y git curl ca-certificates gnupg lsb-release locales locales-all nano jq wget

# Setting Locales
ENV LC_ALL en_US.UTF-8
ENV LANG en_US.UTF-8
ENV LANGUAGE en_US.UTF-8

# Install Pyenv and Python
ENV PYENV_ROOT="/usr/local/pyenv" PYENV_GIT_TAG=v${PYENV_VERSION}
ENV PATH="${PATH}:${PYENV_ROOT}/bin"
RUN apt-get install -y build-essential libssl-dev zlib1g-dev libbz2-dev libreadline-dev libsqlite3-dev curl libncursesw5-dev xz-utils tk-dev libxml2-dev libxmlsec1-dev libffi-dev liblzma-dev \
    && curl https://pyenv.run | bash \
    && pyenv install ${PYTHON_VERSION} \
    && pyenv global ${PYTHON_VERSION}

# Install Poetry
RUN unset PYENV_VERSION && pyenv exec pip install poetry==${POETRY_VERSION}

Lifecycle Scripts

You might have noticed the snippet in the Dockerfile above

# Copy custom scripts and set the required permissions
COPY custom-scripts/ /tmp/scripts/
RUN chmod +x /tmp/scripts/*

What it does is to copy some files from the custom-scripts folder located inside the .devcontainer and give the necessary permissions for them to be executed. This will be useful later on where we are going to run these scripts in some phases of the Docker lifecycle (e.g. when it creates, every time it starts and so on).

Step 03: The devcontainer.json file

The devcontainer.json file is where you specify configuration for VSCode to run your containers. You can see the complete specification of settings and metadata on Development Containers metadata reference.

{
    "name": "DevContainer",
    "build": {
      "dockerfile": "Dockerfile",
      "args": {
          "PYENV_VERSION": "2.3.18",
          "POETRY_VERSION": "1.5.1",
          "PYTHON_VERSION": "3.11.3"
      }
    },
    "mounts": [
      "source=/var/run/docker.sock,target=/var/run/docker.sock,type=bind,consistency=default"
    ],
    "postStartCommand": "/bin/bash /tmp/scripts/poststart.sh",
    "customizations": {
      "vscode": {
        "extensions": [
          "ms-python.vscode-pylance",
          "njpwerner.autodocstring",
          "bungcip.better-toml",
          "ms-python.black-formatter"
        ],
        "settings": {
          "editor.rulers": [79, 110],
        }
      }
    }
}

Build

The build information is responsible to setup the information for VSCode to build your container. This includes args, target, caches, the path to the Dockerfile and so on. The snippet below defines the path of the Dockerfile and set the args. See more on Image or Dockerfile specific properties.

{
  ...
  "build": {
    "dockerfile": "Dockerfile",
    "args": {
      "PYENV_VERSION": "2.3.18",
      "POETRY_VERSION": "1.5.1",
      "PYTHON_VERSION": "3.11.3"
    }
  },
  ...
}

postStartCommand and Lifecycle Scripts

Lifecycle scripts is a feature by DevContainers where you can make some commands run in some phases of the Docker Lifecycle. For example, on our example we use the postStartCommand that runs everytime we start the container. This is useful for example to always update your OS or to make all your python dependencies installed. But there are more, you can use for example onCreateCommand that will execute only when the container is created. You can read more about this on Lifecycle scripts and Start a process when the container starts.

{
  ...
  "postStartCommand": "/bin/bash /tmp/scripts/poststart.sh",
  ...
}

Customizations

The customizations property is where DevContainers define information specific for your IDE. For VSCode we have two properties vscode.settings and vscode.extensions that defines the default settings and installed extensions for the environment. The example devcontainer.json adds some extensions such as pylance, autodocstring, better-toml and black-formatter to the container and sets two editor rules for 79 (PEP8) and 110 characters.

{
  ...
   "customizations": {
      "vscode": {
        "extensions": [
          "ms-python.vscode-pylance",
          "njpwerner.autodocstring",
          "bungcip.better-toml",
          "ms-python.black-formatter"
        ],
        "settings": {
          "editor.rulers": [79, 110],
        }
      }
    }
}

Some tips for making this configuration easier:

For extensions, you can go to any extension in the tab, expand the options on the ... (three dots) and click on the option add to devcontainer.json that the file will be automatically managed.
For settings you can open the command pallet on Ctrl + Shift + P and go to any configuration that you wish to alter, click on the More actions represented by a gear icon, copy setting as json and add this to the json file.

Mounts (and Docker-From-Docker)

Mounts is the metadata used to specify your container's volumes as in any docker container. A common example of mount that is used is when you want to run docker within your container. Here we are setting up the Docker-from-Docker (or Docker-outside-of-Docker) approach where we mount the host's docker socket into the container so any build or docker command will actually execute on the host. See more on General devcontainer.json properties

{
  ...
  "mounts": [
    "source=/var/run/docker.sock,target=/var/run/docker.sock,type=bind,consistency=default"
  ],
  ...
}

Personalizing your environment with dotfiles

Each Linux user has their own set of settings for their software, some use a personalized shell like zshell, other use Vim with some tweaks as their main editor. To maintain the productivity of everyone it is important to maintain these settings across environments. Many Linux software uses dotfiles as their main way to store their settings and this is what we are going to explore.

One of the features of the DevContainer extension on VSCode is that we can pull dotfiles directly from a Github repository (public or private). This can be done by adding the following settings on your VSCode Settings (JSON) (or looking for similar settings in the UI).

{
  "dotfiles.repository": "your-github-id/your-dotfiles-repo", 
  "dotfiles.targetPath": "~/dotfiles",
  "dotfiles.installCommand": "~/dotfiles/install.sh"
}

What this means is that VSCode will copy all the content of the dotfiles.repository into the folder dotfiles.targetPath inside the container and will execute the dotfiles.installCommand. You can omit the installCommand and it will default to a install.sh script inside the targetpath. You can read more on
Personalizing with dotfile repositories.

A real example of a install.sh (used by me) is presented below. This script will install zshell, Powerlevel10K theme and create symbolic links of my personalized settings into the required ones in the system.

#!/usr/bin/env sh
set -e

DOTFILES_LOCATION="${HOME}/dotfiles"    
export DOTFILES_LOCATION;

apt install -y zsh fonts-powerline fzf

if [ -d "${HOME}/.oh-my-zsh" ]; then
  printf "oh-my-zsh is already installed\n"
else
  sh -c "$(curl -fsSL https://raw.githubusercontent.com/ohmyzsh/ohmyzsh/master/tools/install.sh)" "" --unattended # Zshell
  git clone --depth=1 https://github.com/romkatv/powerlevel10k.git ${ZSH_CUSTOM:-$HOME/.oh-my-zsh/custom}/themes/powerlevel10k
fi

echo "creating symlinks for zsh"
ln -sf "${DOTFILES_LOCATION}/zsh/.zshrc" "${HOME}/.zshrc"
ln -sf "${DOTFILES_LOCATION}/zsh/.p10k.zsh" "${HOME}/.p10k.zsh"
ln -sf "${DOTFILES_LOCATION}/zsh/.zprofile.zsh" "${HOME}/.zprofile.zsh"

Note that everything in this section is completely optional, and you can use the default configuration for each software.

Conclusions

Learn about DevContainers were a turning point in the projects that I worked with reducing problems related to manual configuration of a series of tools on different environments. This tutorial should help you achieve the same benefits as I had by setting your custom development environment.

Thanks for Reading! Please, feel free to drop any suggestions, discussions, tips on the section below!