DEV Community: Daniele Salatti

Goals for 2021: Learning Go and Rust

Daniele Salatti — Thu, 31 Dec 2020 05:03:02 +0000

I'll keep this short. In the past 6 or so years in Amazon I've been mostly working with Java and sometimes a little bit of Python. There's nothing wrong with that. I'm not against any of these languages, I might even enjoy working with them - at least when people don't abuse design patterns just for the sake of it and use 75 levels of indirection because "best practices", ending up with code that is impossible to maintain. And unlike what some think, I believe both languages are going to be around for a long time - their market share is far too big and they are far too widespread to be disappearing any time soon.

But I'm digressing and I said I was going to keep this short.

Like many other developers I like to spend time researching new technologies, look at new programming languages, and just experiment with things in my free time. Over the years two languages caught my attention, and I figured this is as good a time as any to start picking up on a couple of them. I'm talking about Go and Rust.

Now, to be honest: I hate new years resolution, and I don't believe they make much sense. It's something you commit to and then just drop a few weeks later... (fun fact: I do have some sort of yearly cycle analysis that I do regularly - e.g. around finances - but I use my birthday as a "fiscal year" of sorts). I do want to learn these two new (to me) languages though, and I figured I'd try to use this blog as an "accountability buddy". So here we go.

I'll be posting my notes and progress while I try to learn Go and Rust, in a format that can be used by others to learn as I go. If it ends up being decent enough it might even become a little course in Go and Rust programming. And I will release anything I end up building while learning as open source on my GitHub profile.

So there it is. Stay tuned and follow along :)

Monitoring With Prometheus: Experiments With a Raspberry Pi

Daniele Salatti — Wed, 26 Aug 2020 06:55:38 +0000

Code available on GitHub here: DanieleSalatti/Prometeo

What Is Prometheus

Prometheus is cool software used for event monitoring and alerting. It includes a time series database used to record real-time metrics, it has a pretty flexible query language and alerting functions. It scrapes metrics over HTTP using a pull model - which has some advantages in terms of scaling compared to a push model (more on this in a future post).

Requirements

Docker on a PC or laptop
A Raspberry Pi (or ARMv6/v7 boards)

You can get a Raspberry Pi from here if you don't have one already:

The links above are for the Raspberry Pi only. You'll also need a power source, an SD card, a micro-HDMI to HDMI adapter, and optionally an enclosure. Sometimes it's worth to get it all as a bundle, so here's a couple links:

I have the 8GB kit from the last link and can vouch for them.

Preparing the Docker Image

Well, actually, before we build our Docker image we need a way to test our Prometheus instance when we run it locally. That's a pretty simple thing to do as all we need it to set up a NodeExporterinstance. A NodeExporter is a simple local service that exposes an HTTP endpoint (usually on port 9100) and publishes a set of system/hardware metrics, allowing Prometheus to scrape it.

I am going to set up my NodeExporter on my Raspberry Pi(s), but you can set it up locally or on any instance/VPS you want to monitor.

Set up a NodeExporter Instance

Let's start by installing a NodeExporter:

 pi@televisione:~ $ sudo apt-get install prometheus-node-exporter

Once the installation process finishes, we can test out endpoint:

pi@televisione:~ $ curl localhost:9100/metrics | less

You should see a bunch of metrics being emitted. Done.

Create a New Docker Image

Thankfully for us there already is a Prometheus image on Docker Hub, so all we need to do is change the config file to suit our needs.

Extract default config file from Docker image:

$ docker create --name prom_empty prom/prometheus
$ docker cp prom_empty:/etc/prometheus/prometheus.yml ./prometheus.yml

Use your favorite editor and open ./prometheus.yml. You'll see a block similar to this:

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
    - targets: ['localhost:9090']

We are going to add our endpoints in the scrape_configs section. So let's add our test endpoint (comments removed for readability):

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
    - targets: ['localhost:9090']
  - job_name: 'televisione'
    static_configs:
      - targets: ['192.168.1.224:9100']

The second job name is called "televisione" because that Raspberry Pi is attached to my smart TV (TV = televisione in Italian). Feel free to change that name. Now we need to create our Dockerfile, telling Docker to copy our config file over for Prometheus to use. Here's the content:

FROM prom/prometheus
COPY ./prometheus.yml /etc/prometheus/prometheus.yml

And finally we need to build our image:

$ docker build -t prometheus/cluster-local .

Once the command succeed we can run Prometheus:

$ docker run -p 9091:9090 --restart=always --name prometheus-local -d prometheus/cluster-local

Note that I'm redirecting port 9091 from the host to port 9090 on the container. That's because I am going to have another Prometheus instance running on that Raspberry Pi as part of a separate tool that I'll discuss in a follow up post. You can leave it as 9090:9090 if you prefer.

When you want to stop the container type in: docker rm -f prometheus-local. Remember that if you make any change to the config file, you will need to build the image again, then stop the container and run it again as above.

Now you can go to http://localhost:9091/ and see Prometheus up and running. Click on Status -> Targets in the top menu to see a list of targets. There should be only two at the moment:

The local Prometheus instance and
The test NodeExporter we setup earlier

Deploying to the Raspberry Pi

Install Docker on your Raspberry Pis. Instructions here will differ depending on the OS. I'll cover Raspbian and Ubuntu server, since that's what I have.

Install Docker on Ubuntu Server

Do not use the install script, do it properly. Instructions here and below in a shortened version.

In essence what we have to do is install a few dependencies first, then add the Docker repository to our repository list, and finally install Docker.

$ sudo apt-get update

$ sudo apt-get install \
    apt-transport-https \
    ca-certificates \
    curl \
    gnupg-agent \
    software-properties-common

 $ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

Remember to always validate any key you get from the Internet. You can do so by looking for the last 8 digits of the fingerprint:

$ sudo apt-key fingerprint 0EBFCD88

pub rsa4096 2017-02-22 [SCEA]
      9DC8 5822 9FC7 DD38 854A E2D8 8D81 803C 0EBF CD88
uid [unknown] Docker Release (CE deb) <docker@docker.com>
sub rsa4096 2017-02-22 [S]

If the fingerprint looks like this one you are good to go: 9DC8 5822 9FC7 DD38 854A E2D8 8D81 803C 0EBF CD88

I have installed Ubuntu 64 bit on my Raspberry Pis 4 (8 GB RAM), so to add the repository I need to type this:

sudo add-apt-repository \
   "deb [arch=arm64] https://download.docker.com/linux/ubuntu \
   $(lsb_release -cs) \
   stable"

If you have a different architecture you need to replace arm64 with armhf or amd64 depending on what you have.

And finally:

 $ sudo apt-get update
 $ sudo apt-get install docker-ce docker-ce-cli containerd.io

Install Docker on Raspbian

Unfortunately with Raspbian we cannot use the repository, so we need to use the convenience script:

$ curl -fsSL https://get.docker.com -o get-docker.sh
$ sudo sh get-docker.sh

Start Prometheus

Now that we have Docker running, we can copy our config and Docker file over to the Raspberry Pi and start our container. For convenience, I uploaded everything we need in this GitHub repository: https://github.com/DanieleSalatti/Prometeo

If you do check that out, remember to change your config files. It would probably be a good idea to create a fork, so you can track your own changes there.

Once everything is moved over to the Pi, all that's needed is to build and run our image again:

$ docker build -t prometheus/cluster-local .
$ docker run -p 9091:9090 --restart=always --name prometheus-local -d prometheus/cluster-local

Data Persistence

Docker volumes aren't the best way to persist data, so I added a little script in the GitHub repository to start our Prometheus image in a slightly different way. The script is called run.sh. Let's quickly take a look:

#! /bin/bash
mkdir -p /media/usb-ssd-1/prometeo # creates a folder for your data
ID=$(id -u) # saves your user id in the ID variable

docker run -p 9091:9090 --restart=always \
--user $ID \
--volume "/media/usb-ssd-1/prometeo:/prometheus" \
--name prometheus-local -d prometheus/cluster-local

As you can see, at the top of the file we create a new folder (if it doesn't already exist - the -p parameter takes care of that). After that we start our image, but there's one additional parameter that we pass to docker run: --volume "/media/usb-ssd-1/prometeo:/prometheus". That allows us to specify a mount-point on our host for a folder in our container. We also run our image with our own user, and that's to ensure that we have the correct read/write permissions for the folder we just created. Be sure to change the run.sh file to suit your needs before attempting to start it.

Grafana

Little bonus: in the GitHub repository I also included a second little script to run a Grafana image. No additional config is needed since Grafana can be configured from the UI itself. Just like the previous script it will attempt to create a folder and start the image with a volume mapped to that folder for data persistence. Be sure to tweak it as needed.

Oh, and be sure to import this dashboard once you start playing with your data. It's pretty well done.

What About Alerting?

I'm not going to look at the alerting section for now, but given that this will be a series remember to tune in and check for updates. Better yet, subscribe :)

Alright, that's it for now.

Docker: permission denied while trying to connect to the Docker daemon socket

Daniele Salatti — Sun, 23 Aug 2020 16:19:02 +0000

This is mostly a memo for myself since I'm sure I'll stumble onto this in the future as well, but I figured it might be useful for others as well.

If you try to run a docker command you may get an error like the following:

Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post http://%2Fvar%2Frun%2Fdocker.sock/v1.40/containers/create?name=prom_empty: dial unix /var/run/docker.sock: connect: permission denied

That is probably because your user is not in the docker group.

To fix this, simply create the group and add yourself to it:

$ sudo groupadd docker
$ sudo usermod -aG docker $USER
$ newgrp docker

If you installed Docker using snap, i.e. with something like:

$ sudo snap install docker

Then you will need a couple additional steps:

$ sudo snap disable docker
$ sudo snap enable docker

Log out and back in again, then test your Docker installation by running the hello-world image:

$ docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
0e03bdcc26d7: Pull complete 
Digest: sha256:7f0a9f93b4aa3022c3a4c147a449bf11e0941a1fd0bf4a8e6c9408b2600777c5
Status: Downloaded newer image for hello-world:latest

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (amd64)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
 https://hub.docker.com/

For more examples and ideas, visit:
 https://docs.docker.com/get-started/

More details and useful post-install resources here.

Alright, that was it.