DEV Community: Stepan Vrany

Privacy matters: send instant messages without providing any personal details!

Stepan Vrany — Sun, 20 Mar 2022 15:28:32 +0000

You might know WhatsApp, Telegram or Signal. These messengers share one concept - they use your cell phone number as the primary identifier. Moreover, you can't create an account without the cell phone number.

Do I need to care about cell phone number?

Technically it's possible to get totally anonymous cell phone number. You can use so-called prepaid (e)SIM cards together with anonymous or pseudonymous payment methods.

In some countries you can even purchase "physical" SIM cards personally in supermarket. For instance, in Czechia it's still legal but this service is slowly vanishing all over the world. It's the same story, governments fight the terrorism.

But the (legal) ownership of the cell phone number is just one part of the story. The next part of the story is traceability, in some circumstances service providers have to cooperate with regime and then it's just a piece of cake to reveal you location. Busted.

So let's avoid cell phone number

So our best choice is just don't use services that requires cell phone number. Problem solved.

Well not quite. It's pretty hard to find such services nowadays. Everyone wants to know your number, even the email providers. I don't know the reason why it is like that but that's the reality and we need to cope with that.

But I have good news, such providers do exist! There's just handful of them but they do exist! So let's look at them.

Threema

Threema is Switzerland-based instant messaging app with strong focus on privacy. Providing your cell phone number is optional so it perfectly fits to our list.

In Slovakia, it has bad reputation due to Jan Kuciak`s case but it has nothing to do with the Threema itself. It's rather about the compromised device. So Threema can be still considered as the more private alternative. Also check the list of data collected by the provider.

Matrix

Another interesting project is Matrix. It's using concept of homeservers so you have to find your homeserver (that does not require email address), create an account there and then you can freely communicate.

If you're concerned about the data collected by the homeservers - check this document. Basically you can be tracked by some identifiers so you have to use this tool with caution together with other techniques to hide your identity. But it does not mean it's a bad tool. It's just about how you use this tool.

Session

This one is my personal favorite. It's easy to use, it does not need cell phone number and it does not collect any personal data.

One major feature of session is the structure of the network - every message is transported through a decentralized onion routing network to obfuscate your identity as much as possible.

Session also don't have any centralized identity management so you don't need to remember any username or password. Instead, you have to write down mnemonic seed that can be used to recover your identity on any device. This process is well-known in the world of crypto currencies.

Enough of talking, check the FAQ to get more information about this tool! As I've mentioned before, this is my personal favorite so that's why I went through some features in detail.

Wrap

And that's all I was able to find. Unfortunately. In past 10 years we cared more about the convenience than privacy so here we are. Thankfully we have at least some tools so we need to take this opportunity, reclaim our privacy and make the other vendors to deliver similar features.

And again, my personal favorite is Session due to extremely easy usage.

It's very important to be able to communicate privately and hide our identity if needed. If no one can't determine our identity from the data collected - we can't be persecuted. Now nor in the future.

And again, don't forget that tool works as intended only if it's used correctly. Even the most secure tool can reveal your identity and trust me there are bad actors that know how to abuse such information, especially those controlling monopoly on violence. Read the documentation. Read the FAQ. Read reddit discussions. Stay safe.

If I could conclude with a one sentence, it would be: try to avoid services requiring your cell phone number.

Cheers!

Getting started with Tor hidden services

Stepan Vrany — Wed, 02 Mar 2022 18:47:38 +0000

This will be happening more and more frequently in the online world: people get silenced. If you're a ruler, it's tempting to selectively deliver only certain information and let's face it - surface web is perfectly equipped for such scenarios. Providers of services have too much to lose if disobey. This is just how power works. But that's a bit different topic...

Today we're gonna talk about the world without centralised authorities, the world of Tor hidden services.

Enter the .onion!

In Tor we don't use standard domain names like cz or eu. We use specialised .onion generic name that is resolvable only in Tor network. Compared to standard domain names it has one major difference beyond the scope of the visibility - there's no vendor where you have to purchase a name for you service.

This is how it works - you create a hidden service, a new keypair is generated and part of the public key is then used as your .onion name. You can read more about the mechanism in this post.

Please read previous 2 paragraphs again - there's no authority that's responsible for the allocation of names. And when there's no authority - you can't approach it and make it obey with the force 👊

Then you just point Tor daemon to your webserver and voila - everyone with Tor browser (or alternative configuration with SOCKS5 proxy) can access your hidden service!

Now let's create our first hidden service. That's gonna be fun.

Part I: server

We need some server first. It can be anything, you can use tiny Raspberry Pi or you can purchase some VPS from Public Cloud providers. I don't have any specific recommendations here. Server in your homelab is more decentralized, on the other hand VPS is easy to obtain. It's up to you.

I'm gonna use VPS purchased from Digital Ocean pre-installed with the latest LTS Ubuntu.

In a couple of seconds I can connect with ssh there and proceed with the configuration!

Part II: application

In this tutorial we'll be serving some dummy content from the application written in Go. For the sake of simplicity I'm gonna use some snippets from the Fiber framework's homepage. So here we go, this is our application.

package main

import "github.com/gofiber/fiber/v2"

func main() {
    app := fiber.New()

    app.Get("/", func(c *fiber.Ctx) error {
        return c.SendString("Hello, World 👋!")
    })

    app.Listen(":3000")
}

Once again, this is Go but you can write your application in any available tool. And of you want to serve static content - use Caddy or Nginx.

So now I'm gonna build and test the app:

go mod init main
go mod tidy
go build main.go
./main&
curl localhost:3000

This is what we get as the output:

Hello, World 👋!

Part III: supervision

In the modern operating system we can always use some way how to supervise your applications. This is extremely important and I'm gonna give you one specific example: when your server restarts, you want to make sure that applications starts together with the server.

In the case we use Ubuntu so systemd service unit needs to configured.

So let's move the application to some well-known path

mkdir /opt/app
mv main /opt/app/main

and create the systemd service unit in /etc/systemd/system/app.service file:

[Unit]
Description=App
After=network.target

[Service]
Type=simple
ExecStart=/opt/app/main

[Install]
WantedBy=multi-user.target

Now we can enable and start the service with systemctl commands:

systemctl enable app
systemctl start app

Status of the service can be verified by invoking systemctl status app:

● app.service - App
     Loaded: loaded (/etc/systemd/system/app.service; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2022-03-02 18:08:29 UTC; 2s ago
   Main PID: 2289 (main)
      Tasks: 4 (limit: 1132)
     Memory: 1.3M
     CGroup: /system.slice/app.service
             └─2289 /opt/app/main

Part IV: Tor daemon

All the stuff we were doing till now was pretty much the same as you'd do for standard surface web app, right? So let's make it special. To do so we need to install Tor daemon first.

In this Ubuntu 20.04 we just need to create a new apt source in /etc/apt/sources.list.d/tor.list

deb     [signed-by=/usr/share/keyrings/tor-archive-keyring.gpg] https://deb.torproject.org/torproject.org focal main
deb-src [signed-by=/usr/share/keyrings/tor-archive-keyring.gpg] https://deb.torproject.org/torproject.org focal main

add public key to the keyring

wget -qO- https://deb.torproject.org/torproject.org/A3C4F0F979CAA22CDBA8F512EE8CBC9E886DDD89.asc | gpg --dearmor | tee /usr/share/keyrings/tor-archive-keyring.gpg >/dev/null

and install the package:

apt-get update
apt-get install tor -y

If you're using different operating system - check out the official documentation. You know the drill I guess 😁

As the result we have Tor daemon running!

systemctl status tor
● tor.service - Anonymizing overlay network for TCP (multi-instance-master)
     Loaded: loaded (/lib/systemd/system/tor.service; enabled; vendor preset: enabled)
     Active: active (exited) since Wed 2022-03-02 18:16:30 UTC; 1min 28s ago
   Main PID: 2959 (code=exited, status=0/SUCCESS)
      Tasks: 0 (limit: 1132)
     Memory: 0B
     CGroup: /system.slice/tor.service

Part V: pointing Tor to our app

And this is the very last part of this tutorial. Tor just needs to be configured to "advertise" our application to the network.

Open the main configuration file /etc/tor/torrc and add following lines to the respective part of the configuration.

HiddenServiceDir /var/lib/tor/app/
HiddenServicePort 80 127.0.0.1:3000

The second line basically says that incoming traffic on port 80 should be routed to localhost:300. And this is the port where the simple app listens.

Tor daemon needs to be restarted first with systemctl restart tor command and whent it's up again we can inspect /var/lib/tor/app/ directory - the home directory of our hidden service.

ls /var/lib/tor/app/ -lah
total 24K
drwx--S--- 3 debian-tor debian-tor 4.0K Mar  2 18:22 .
drwx--S--- 4 debian-tor debian-tor 4.0K Mar  2 18:23 ..
drwx--S--- 2 debian-tor debian-tor 4.0K Mar  2 18:22 authorized_clients
-rw------- 1 debian-tor debian-tor   63 Mar  2 18:22 hostname
-rw------- 1 debian-tor debian-tor   64 Mar  2 18:22 hs_ed25519_public_key
-rw------- 1 debian-tor debian-tor   96 Mar  2 18:22 hs_ed25519_secret_key

The last 2 lines are interesting, this is the key pair I was talking about. Please note that private key needs to be kept in secret, anyone with access to this key can use your onion name for some nasty s**t! And when we talk about the name, let's just check the contents of the hostname file.

cat /var/lib/tor/app/hostname
ocig6kqiahyictb5y7o37ympfragi4nlzimuylwsfco76ksi2l5edvyd.onion

And this is, ladies and gentlemen, a domain name we can use from Tor browser. Let's do it.

Result

Install Tor browser or Brave (Brave needs to be launched in specialised Tor mode) and enter onion name to the address bar.

This is the service we were working on! It looks just like any other web application but it has one major feature - putting this down requires much more effort that the surface web equivalent.

Now you can share your fantastic content even with the folks who lives in less free countries 🔥🔥🔥

Do you need any help with Tor hidden service? Don't hesitate to contact me. I'm 100% committed to make world a better place through the free speech and better privacy ❤️

Title image: https://www.deviantart.com/wuphaz/art/Tor-wallpaper-682786904

Part IV: Telegram notifications

Stepan Vrany — Wed, 23 Feb 2022 10:17:22 +0000

When we have all the monitoring component in the place we might be wondering how to receive notification when something went wrong. Good question.

Alertmanager does the trick and it natively supports all the major channels - Slack, PagerDuty, OpsGenie and more. But it does not support Telegram which is kinda popular even for the business communication in smaller companies.

Let's fix it with a few AWS resources and simple Go lambda function.

SNS

SNS is one of the supported Alermanager's channels. It's actually very convenient in this case since we have the IAM role attached to instance - we don't need to hassle with credentials.

Let's dive into Terraform again.



resource "aws_sns_topic" "prometheus_alerts" {
  name = "prometheus_alerts"
}

SQS

The next part is the queue, we can possibly connect a lambda directly do SNS but SQS gives us more reliability. SQS will effectively become the only SNS subscriber and messages will be queued for the further processing.



resource "aws_sqs_queue" "prometheus_alerts" {
  name = "prometheus_alerts"
}

And since SNS topic will be pushing messages to this queue - we also need to allow this:



resource "aws_sqs_queue_policy" "prometheus_alerts" {
  queue_url = aws_sqs_queue.prometheus_alerts.id

  policy = <<POLICY
{
  "Version": "2012-10-17",
  "Id": "sqspolicy",
  "Statement": [
    {
      "Sid": "First",
      "Effect": "Allow",
      "Principal": "*",
      "Action": "sqs:SendMessage",
      "Resource": "${aws_sqs_queue.prometheus_alerts.arn}",
      "Condition": {
        "ArnEquals": {
          "aws:SourceArn": "${aws_sns_topic.prometheus_alerts.arn}"
        }
      }
    }
  ]
}
POLICY
}

Also, create the subscription:



resource "aws_sns_topic_subscription" "prometheus_alerts" {
  topic_arn = aws_sns_topic.prometheus_alerts.arn
  protocol  = "sqs"
  endpoint  = aws_sqs_queue.prometheus_alerts.arn
}

Lambda function

Here we go again, Lambda needs role, permissions.. so let's create these resources first.



data "aws_caller_identity" "current" {}
data "aws_region" "current" {}

resource "aws_iam_role" "alertmanager_notify" {
  name = "alertmanager_notify"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Principal = {
          Service = "lambda.amazonaws.com"
        }
        Action = "sts:AssumeRole"
      }
    ]
  })
}

resource "aws_iam_policy" "alertmanager_notify" {
  name = "alertmanager_notify"
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Action = [
          "logs:CreateLogGroup",
        ]
        Resource = [
          "arn:aws:logs:${data.aws_region.current.name}:${data.aws_caller_identity.current.account_id}:log-group:/aws/lambda/alertmanager_notify",
        ]
      },
      {
        Effect = "Allow"
        Action = [
          "logs:CreateLogStream",
          "logs:PutLogEvents",
        ]
        Resource = [
          "arn:aws:logs:${data.aws_region.current.name}:${data.aws_caller_identity.current.account_id}:log-group:/aws/lambda/alertmanager_notify:*",
        ]
      },
      {
        "Effect": "Allow",
        "Action": [
          "sqs:DeleteMessage",
          "sqs:GetQueueAttributes",
          "sqs:ReceiveMessage",
        ],
        "Resource": [
          aws_sqs_queue.prometheus_alerts.arn,
        ]
      },
    ]
  })
}

resource "aws_iam_role_policy_attachment" "alertmanager_notify" {
  role       = aws_iam_role.alertmanager_notify.name
  policy_arn = aws_iam_policy.alertmanager_notify.arn
}

With such permissions, Lambda function is able to process messages from the SQS queue and delete them when processed. Cloudwatch part is pretty self-descriptive - we want to see some logs. Lambda function definition itself is straightforward:



resource "aws_lambda_function" "alertmanager_notify" {
  filename         = "${path.module}/assets/function.zip"
  source_code_hash = filebase64sha256("${path.module}/assets/function.zip")
  function_name    = "alertmanager_notify"
  role             = aws_iam_role.this.arn
  handler          = "main"
  runtime          = "go1.x"

  lifecycle {
    ignore_changes = [
      filename,
      last_modified,
      source_code_hash,
    ]
  }

  environment {
    variables = {
      TELEGRAM_TOKEN   = <telegram token from bot father>
      TELEGRAM_CHANNEL = <telegram group id>
    }
  }
}

Note the ${path.module}/assets/function.zip, this is a zip file with some dummy code. We only need this archive for the initial creation. The final code will be pushed to the AWS externally.

Lambda code

For the integration with Telegram we're gonna use github.com/go-telegram-bot-api/telegram-bot-api library. It's just thin wrapper over the Telegram API and it's sufficient for this use case.

Please note that signatures of handle and sendMessage should ideally contain interface, some sort of TelegramSender or so. But for the sake of simplicity we use BotAPI directly 😊



package main

import (
    "context"
    "fmt"
    "github.com/aws/aws-lambda-go/events"
    "github.com/aws/aws-lambda-go/lambda"
    tgbotapi "github.com/go-telegram-bot-api/telegram-bot-api"
    log "github.com/sirupsen/logrus"
    "os"
    "strconv"
)

var (
    telegramToken   = os.Getenv("TELEGRAM_TOKEN")
    telegramChannel = os.Getenv("TELEGRAM_CHANNEL")
    parseMode       = "HTML"
)

func init() {
    log.SetFormatter(&log.JSONFormatter{})
}

func sendMessage(bot *tgbotapi.BotAPI, channel int64, message string, mode string) error {
    log.Info("sending notification to Telegram")
    msg := tgbotapi.NewMessage(channel, message)
    msg.ParseMode = mode

    _, err := bot.Send(msg)
    if err != nil {
        return fmt.Errorf("could not send message: %s", err)
    }

    return nil
}

func handle(bot *tgbotapi.BotAPI, channel int64) func(context.Context, events.SQSEvent) error {
    return func(ctx context.Context, event events.SQSEvent) error {
        for _, record := range event.Records {
            log.
                WithField("message_id", record.MessageId).
                Info("processing SQS record")
            err := sendMessage(bot, channel, record.Body, parseMode)
            if err != nil {
                log.Fatalf("could process message: %s", err)
            }

        }
        return nil
    }
}

func main() {
    // create telegram client
    bot, err := tgbotapi.NewBotAPI(telegramToken)
    if err != nil {
        log.Fatalf("could not create telegram client")
    }

    // parse
    telegramChannelInt, err := strconv.ParseInt(telegramChannel, 10, 64)
    if err != nil {
        log.Fatalf("could not parse channel id")
    }

    lambda.Start(handle(bot, telegramChannelInt))
}

Let's deploy the function!



GOARCH=amd64 GOOS=linux go build main.go
zip function.zip main
aws lambda update-function-code --function-name alertmanager_notify --zip-file fileb://./function.zip

Alertmanager configuration

First of all, we need to allow interaction with SNS topic for the instance where the Alertmanager is running. Add this statement to the IAM policy from the first chapter:



      {
        Effect = "Allow"
        Action = [
          "sns:Publish",
        ]
        Resource = [
          aws_sns_topic.prometheus_alerts.arn,
        ]
      },

The last bit is yaml for the Alertmanager. This is perhaps the simplest configuration:



global:

  resolve_timeout: 1m

receivers:

- name: sns

  sns_configs:

  - message: |

      {{ if eq .Status "firing" }}🔥 {{ end }}{{ if eq .Status "resolved" }}✅ {{ end }}[{{ .Status | toUpper }}] {{ .CommonLabels.alertname }}

      {{ range .Alerts }}

          <b>Alert:</b> {{ .Annotations.title }}{{ if .Labels.severity }} - {{ .Labels.severity }}{{ end }}

          <b>Description:</b> {{ .Annotations.description }}

          <b>Details:</b>

          {{ range .Labels.SortedPairs }}- {{ .Name }}: <i>{{ .Value }}</i>

          {{ end }}

      {{ end }}

    sigv4:

      region: eu-west-1

    topic_arn: <SNS topic ARN>

route:

  group_by:

  - "..."

  group_interval: 30s

  group_wait: 5s

  receiver: sns

  repeat_interval: 3h

Result

Now we're able to receive alerts in the Telegram group, see the following example that comes from the similar configuration. Pretty neat, huh?

Wrap

And this is the end of this series. I've been working on this setup in past two months and I must say I'm pretty happy with the overall result.

Here's also one final disclaimer - do not forget to establish external monitoring for the Prometheus. Expose readiness server's probe to the internet and use tools such uptime robot because you really want to know when Prometheus goes down.

Do you have any questions? Ping me on twitter or ask it directly here. I'll try to do my best to help you.

Part III: Grafana

Stepan Vrany — Wed, 23 Feb 2022 09:06:02 +0000

So now we have the Prometheus up and running and we even send some metrics there thanks to Grafana cloud agent. How to view these metrics in some more convenient way?

Yes, the answer is Grafana, the best in class tool for the job. And since we have a Kubernetes cluster - we're gonna run it inside this cluster. Grafana is not really resource-intensive beast and this will be also cheaper than managed services like Amazon Managed Grafana.

Persistence

Grafana needs some persistent store for users, dashboards and other stuff. We can use Persistent Volume Claim but again - we want to keep our cluster lean. So our next choice is RDS in its smallest configuration.

data "aws_caller_identity" "current" {}
data "aws_region" "current" {}

resource "random_password" "password" {
  length  = 32
  special = false
}

resource "aws_security_group" "this" {
  name   = "grafana_${var.environment}"
  vpc_id = var.vpc_id
}

resource "aws_security_group_rule" "egress_all" {
  type     = "egress"
  to_port  = 0
  protocol = "-1"
  cidr_blocks = [
    "0.0.0.0/0",
  ]
  from_port         = 0
  security_group_id = aws_security_group.this.id
}

resource "aws_security_group_rule" "ingress_posgres" {
  type                     = "ingress"
  to_port                  = 5432
  protocol                 = "tcp"
  source_security_group_id = var.source_security_group_id
  from_port                = 5432
  security_group_id        = aws_security_group.this.id
}

resource "aws_db_subnet_group" "this" {
  name       = "grafana-${var.environment}"
  subnet_ids = var.subnet_ids
}

resource "aws_db_instance" "this" {
  availability_zone    = var.availability_zone
  allocated_storage    = 10
  engine               = "postgres"
  engine_version       = var.postgres_engine_version
  instance_class       = var.postgres_instance_class
  identifier           = "grafana-${var.environment}"
  db_name              = "grafana"
  username             = "root"
  password             = random_password.password.result
  skip_final_snapshot  = true
  db_subnet_group_name = aws_db_subnet_group.this.name
  vpc_security_group_ids = [
    aws_security_group.this.id,
  ]
}

You can see here some similarities with the EC2 instance from the first chapter. Once again we've created a separate Security Group with the only ingress rule allowing the communication only from the Kubernetes cluster.

Single sign-on

Grafana supports variety of single sign-on options, in this writeup we're gonna use Gitlab but you can use Github, some generic OIDC solution or the internal database.

Installation of Grafana

Let's prepare values files for the Grafana Helm chart.

replicas: 1

grafana.ini:
  server:
    domain: grafana.<your domain name>
    root_url: https://grafana.<your domain name>/
    enforce_domain: true
    protocol: http
  auth.anonymous:
    enabled: false
  database:
    type: postgres
    host: <RDS endpoint from the previous steps>
    user: root
    password: <RDS password from the previous steps>
    name: grafana
    ssl_mode: require
    max_open_conn: 25
    max_idle_conn: 25
  unified_alerting:
    enabled: true
  alerting:
    enabled: false
  smtp:
    enabled: false
  users:
    auto_assign_org: true
  auth.gitlab:
    enabled: true
    allow_sign_up: true
    client_id: <id from the Gitlab app>
    client_secret: <secret from the Gitlab app>
    scopes: read_api
    auth_url: https://gitlab.com/oauth/authorize
    token_url: https://gitlab.com/oauth/token
    api_url: https://gitlab.com/api/v4
    allowed_groups: <your Gitlab group>

ingress:
  enabled: false

Note the ingress.enabled property. In this particular infrastructure we're using Traefik so we don't need to create Ingress object. Instead I've created a simple IngressRoute resource:

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: grafana
  namespace: grafana
spec:
  entryPoints:
    - websecure
  routes:
  - match: Host(`grafana.<your domain name>`) && PathPrefix(`/`)
    kind: Rule
    priority: 1
    services:
    - name: grafana
      port: 80
      scheme: http

If you're using Nginx ingress controller, then you just need to re-enable it in the values:

ingress:
  enabled: true
  ingressClassName: nginx
  annotations: {}
    kubernetes.io/ingress.class: nginx
  labels: {}
  path: /
  pathType: Prefix
  hosts:
    - grafana.<your domain name>

You know the drill I guess. Let's install this with simple Helm command:

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update
helm upgrade --install grafana grafana/grafana -n grafana -f values.yaml --version 6.21.4

And that's it. For the administration tasks we just need one more thing - admin password:

kubectl get secret --namespace grafana grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo

Adding Prometheus data source to Grafana

This is perhaps the most straightforward task, just log-in, open datasources section in settings and fill in the domain name we've defined in the first chapter.

Wrap

And that's it! In the next chapter I'll show you how to forward alertmanager notifications to Telegram group. Stay tuned!

Part II: Lean Prometheus scraper

Stepan Vrany — Tue, 22 Feb 2022 09:34:13 +0000

In the previous chapter of this series we've created a EC2 instance with Prometheus remote-write collector. So today we're gonna push there some metrics with Grafana cloud agent.

At the moment there's also a possibility to do this with a Prometheus in agent mode. However I did not test this setup yet so we're gonna stick to well-supported Grafana agent.

Kube state metrics

Personally I think that KSM are the most important metrics you need to monitor the cluster. They can show you restarting pods, pending pods and all the states we're trying to avoid somehow.

All you need to do is just install KSM with helm chart:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm install ksm prometheus-community/kube-state-metrics \
  --set image.tag=v2.2.0 \
  --namespace grafana-agent \
  --create-namespace

Grafana cloud agent configuration

Now it's time to prepare configuration file for the scraper. Following snippet comes from the official documentation and it basically covers KSM, cadvisor and Kubelet.

kind: ConfigMap
metadata:
  name: grafana-agent
  namespace: grafana-agent
apiVersion: v1
data:
  agent.yaml: |    
    server:
      http_listen_port: 12345
    metrics:
      wal_directory: /tmp/grafana-agent-wal
      global:
        scrape_interval: 60s
        external_labels:
          cluster: cloud
      configs:
      - name: integrations
        remote_write:
        - url: http://p01.prometheus.local:9090/api/v1/write
        scrape_configs:
        - bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
          job_name: integrations/kubernetes/cadvisor
          kubernetes_sd_configs:
              - role: node
          metric_relabel_configs:
              - source_labels: [__name__]
                regex: container_network_transmit_packets_total|kubelet_certificate_manager_server_ttl_seconds|storage_operation_duration_seconds_bucket|node_namespace_pod_container:container_memory_swap|container_fs_writes_total|container_network_receive_bytes_total|kube_daemonset_status_desired_number_scheduled|cluster:namespace:pod_cpu:active:kube_pod_container_resource_limits|rest_client_requests_total|node_namespace_pod_container:container_memory_working_set_bytes|kubernetes_build_info|kube_node_status_capacity|kubelet_pleg_relist_duration_seconds_bucket|kubelet_running_pods|storage_operation_errors_total|kubelet_running_containers|kube_daemonset_status_number_misscheduled|kube_job_failed|kube_statefulset_status_replicas|kube_job_status_succeeded|container_cpu_cfs_throttled_periods_total|kube_statefulset_status_update_revision|process_resident_memory_bytes|kubelet_pod_start_duration_seconds_count|kubelet_running_container_count|container_fs_writes_bytes_total|machine_memory_bytes|kubelet_cgroup_manager_duration_seconds_count|node_namespace_pod_container:container_memory_rss|kubelet_node_config_error|kubelet_runtime_operations_duration_seconds_bucket|kubelet_pleg_relist_interval_seconds_bucket|kube_job_spec_completions|kube_statefulset_status_current_revision|kube_statefulset_replicas|cluster:namespace:pod_cpu:active:kube_pod_container_resource_requests|kubelet_node_name|kubelet_pod_worker_duration_seconds_bucket|go_goroutines|kubelet_volume_stats_capacity_bytes|kube_horizontalpodautoscaler_status_current_replicas|node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate|kubelet_runtime_operations_errors_total|kube_daemonset_status_number_available|kube_deployment_status_replicas_available|up|storage_operation_duration_seconds_count|kube_daemonset_status_current_number_scheduled|kube_statefulset_status_replicas_updated|kube_node_status_condition|kube_node_status_allocatable|rest_client_request_duration_seconds_bucket|container_cpu_usage_seconds_total|namespace_workload_pod:kube_pod_owner:relabel|kubelet_pleg_relist_duration_seconds_count|kube_pod_owner|namespace_cpu:kube_pod_container_resource_requests:sum|kube_horizontalpodautoscaler_spec_max_replicas|kube_statefulset_status_replicas_ready|container_fs_reads_total|node_namespace_pod_container:container_memory_cache|container_network_transmit_packets_dropped_total|kubelet_volume_stats_inodes_used|kube_node_spec_taint|cluster:namespace:pod_memory:active:kube_pod_container_resource_requests|kube_pod_info|kubelet_cgroup_manager_duration_seconds_bucket|process_cpu_seconds_total|container_memory_cache|kube_statefulset_metadata_generation|kubelet_pod_worker_duration_seconds_count|volume_manager_total_volumes|namespace_cpu:kube_pod_container_resource_limits:sum|kube_deployment_metadata_generation|kube_replicaset_owner|container_memory_swap|kubelet_certificate_manager_client_ttl_seconds|kube_resourcequota|container_fs_reads_bytes_total|kubelet_runtime_operations_total|kube_horizontalpodautoscaler_status_desired_replicas|kube_pod_status_phase|kube_horizontalpodautoscaler_spec_min_replicas|kubelet_server_expiration_renew_errors|kube_pod_container_resource_limits|container_network_transmit_bytes_total|container_network_receive_packets_dropped_total|container_memory_working_set_bytes|kube_pod_container_status_waiting_reason|container_network_receive_packets_total|kube_namespace_created|namespace_workload_pod|kube_pod_container_resource_requests|kubelet_running_pod_count|namespace_memory:kube_pod_container_resource_limits:sum|kube_deployment_status_replicas_updated|kube_statefulset_status_observed_generation|kube_deployment_status_observed_generation|container_cpu_cfs_periods_total|node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile|kubelet_certificate_manager_client_expiration_renew_errors|cluster:namespace:pod_memory:active:kube_pod_container_resource_limits|kube_daemonset_updated_number_scheduled|kubelet_volume_stats_inodes|kube_node_info|kube_deployment_spec_replicas|container_memory_rss|namespace_memory:kube_pod_container_resource_requests:sum|kubelet_volume_stats_available_bytes
                action: keep
          relabel_configs:
              - replacement: kubernetes.default.svc.cluster.local:443
                target_label: __address__
              - regex: (.+)
                replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor
                source_labels:
                  - __meta_kubernetes_node_name
                target_label: __metrics_path__
          scheme: https
          tls_config:
              ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
              insecure_skip_verify: false
              server_name: kubernetes
        - bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
          job_name: integrations/kubernetes/kubelet
          kubernetes_sd_configs:
              - role: node
          metric_relabel_configs:
              - source_labels: [__name__]
                regex: container_network_transmit_packets_total|kubelet_certificate_manager_server_ttl_seconds|storage_operation_duration_seconds_bucket|node_namespace_pod_container:container_memory_swap|container_fs_writes_total|container_network_receive_bytes_total|kube_daemonset_status_desired_number_scheduled|cluster:namespace:pod_cpu:active:kube_pod_container_resource_limits|rest_client_requests_total|node_namespace_pod_container:container_memory_working_set_bytes|kubernetes_build_info|kube_node_status_capacity|kubelet_pleg_relist_duration_seconds_bucket|kubelet_running_pods|storage_operation_errors_total|kubelet_running_containers|kube_daemonset_status_number_misscheduled|kube_job_failed|kube_statefulset_status_replicas|kube_job_status_succeeded|container_cpu_cfs_throttled_periods_total|kube_statefulset_status_update_revision|process_resident_memory_bytes|kubelet_pod_start_duration_seconds_count|kubelet_running_container_count|container_fs_writes_bytes_total|machine_memory_bytes|kubelet_cgroup_manager_duration_seconds_count|node_namespace_pod_container:container_memory_rss|kubelet_node_config_error|kubelet_runtime_operations_duration_seconds_bucket|kubelet_pleg_relist_interval_seconds_bucket|kube_job_spec_completions|kube_statefulset_status_current_revision|kube_statefulset_replicas|cluster:namespace:pod_cpu:active:kube_pod_container_resource_requests|kubelet_node_name|kubelet_pod_worker_duration_seconds_bucket|go_goroutines|kubelet_volume_stats_capacity_bytes|kube_horizontalpodautoscaler_status_current_replicas|node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate|kubelet_runtime_operations_errors_total|kube_daemonset_status_number_available|kube_deployment_status_replicas_available|up|storage_operation_duration_seconds_count|kube_daemonset_status_current_number_scheduled|kube_statefulset_status_replicas_updated|kube_node_status_condition|kube_node_status_allocatable|rest_client_request_duration_seconds_bucket|container_cpu_usage_seconds_total|namespace_workload_pod:kube_pod_owner:relabel|kubelet_pleg_relist_duration_seconds_count|kube_pod_owner|namespace_cpu:kube_pod_container_resource_requests:sum|kube_horizontalpodautoscaler_spec_max_replicas|kube_statefulset_status_replicas_ready|container_fs_reads_total|node_namespace_pod_container:container_memory_cache|container_network_transmit_packets_dropped_total|kubelet_volume_stats_inodes_used|kube_node_spec_taint|cluster:namespace:pod_memory:active:kube_pod_container_resource_requests|kube_pod_info|kubelet_cgroup_manager_duration_seconds_bucket|process_cpu_seconds_total|container_memory_cache|kube_statefulset_metadata_generation|kubelet_pod_worker_duration_seconds_count|volume_manager_total_volumes|namespace_cpu:kube_pod_container_resource_limits:sum|kube_deployment_metadata_generation|kube_replicaset_owner|container_memory_swap|kubelet_certificate_manager_client_ttl_seconds|kube_resourcequota|container_fs_reads_bytes_total|kubelet_runtime_operations_total|kube_horizontalpodautoscaler_status_desired_replicas|kube_pod_status_phase|kube_horizontalpodautoscaler_spec_min_replicas|kubelet_server_expiration_renew_errors|kube_pod_container_resource_limits|container_network_transmit_bytes_total|container_network_receive_packets_dropped_total|container_memory_working_set_bytes|kube_pod_container_status_waiting_reason|container_network_receive_packets_total|kube_namespace_created|namespace_workload_pod|kube_pod_container_resource_requests|kubelet_running_pod_count|namespace_memory:kube_pod_container_resource_limits:sum|kube_deployment_status_replicas_updated|kube_statefulset_status_observed_generation|kube_deployment_status_observed_generation|container_cpu_cfs_periods_total|node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile|kubelet_certificate_manager_client_expiration_renew_errors|cluster:namespace:pod_memory:active:kube_pod_container_resource_limits|kube_daemonset_updated_number_scheduled|kubelet_volume_stats_inodes|kube_node_info|kube_deployment_spec_replicas|container_memory_rss|namespace_memory:kube_pod_container_resource_requests:sum|kubelet_volume_stats_available_bytes
                action: keep
          relabel_configs:
              - replacement: kubernetes.default.svc.cluster.local:443
                target_label: __address__
              - regex: (.+)
                replacement: /api/v1/nodes/${1}/proxy/metrics
                source_labels:
                  - __meta_kubernetes_node_name
                target_label: __metrics_path__
          scheme: https
          tls_config:
              ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
              insecure_skip_verify: false
              server_name: kubernetes
        - job_name: integrations/kubernetes/kube-state-metrics
          kubernetes_sd_configs:
              - role: service
          metric_relabel_configs:
              - source_labels: [__name__]
                regex: container_network_transmit_packets_total|kubelet_certificate_manager_server_ttl_seconds|storage_operation_duration_seconds_bucket|node_namespace_pod_container:container_memory_swap|container_fs_writes_total|container_network_receive_bytes_total|kube_daemonset_status_desired_number_scheduled|cluster:namespace:pod_cpu:active:kube_pod_container_resource_limits|rest_client_requests_total|node_namespace_pod_container:container_memory_working_set_bytes|kubernetes_build_info|kube_node_status_capacity|kubelet_pleg_relist_duration_seconds_bucket|kubelet_running_pods|storage_operation_errors_total|kubelet_running_containers|kube_daemonset_status_number_misscheduled|kube_job_failed|kube_statefulset_status_replicas|kube_job_status_succeeded|container_cpu_cfs_throttled_periods_total|kube_statefulset_status_update_revision|process_resident_memory_bytes|kubelet_pod_start_duration_seconds_count|kubelet_running_container_count|container_fs_writes_bytes_total|machine_memory_bytes|kubelet_cgroup_manager_duration_seconds_count|node_namespace_pod_container:container_memory_rss|kubelet_node_config_error|kubelet_runtime_operations_duration_seconds_bucket|kubelet_pleg_relist_interval_seconds_bucket|kube_job_spec_completions|kube_statefulset_status_current_revision|kube_statefulset_replicas|cluster:namespace:pod_cpu:active:kube_pod_container_resource_requests|kubelet_node_name|kubelet_pod_worker_duration_seconds_bucket|go_goroutines|kubelet_volume_stats_capacity_bytes|kube_horizontalpodautoscaler_status_current_replicas|node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate|kubelet_runtime_operations_errors_total|kube_daemonset_status_number_available|kube_deployment_status_replicas_available|up|storage_operation_duration_seconds_count|kube_daemonset_status_current_number_scheduled|kube_statefulset_status_replicas_updated|kube_node_status_condition|kube_node_status_allocatable|rest_client_request_duration_seconds_bucket|container_cpu_usage_seconds_total|namespace_workload_pod:kube_pod_owner:relabel|kubelet_pleg_relist_duration_seconds_count|kube_pod_owner|namespace_cpu:kube_pod_container_resource_requests:sum|kube_horizontalpodautoscaler_spec_max_replicas|kube_statefulset_status_replicas_ready|container_fs_reads_total|node_namespace_pod_container:container_memory_cache|container_network_transmit_packets_dropped_total|kubelet_volume_stats_inodes_used|kube_node_spec_taint|cluster:namespace:pod_memory:active:kube_pod_container_resource_requests|kube_pod_info|kubelet_cgroup_manager_duration_seconds_bucket|process_cpu_seconds_total|container_memory_cache|kube_statefulset_metadata_generation|kubelet_pod_worker_duration_seconds_count|volume_manager_total_volumes|namespace_cpu:kube_pod_container_resource_limits:sum|kube_deployment_metadata_generation|kube_replicaset_owner|container_memory_swap|kubelet_certificate_manager_client_ttl_seconds|kube_resourcequota|container_fs_reads_bytes_total|kubelet_runtime_operations_total|kube_horizontalpodautoscaler_status_desired_replicas|kube_pod_status_phase|kube_horizontalpodautoscaler_spec_min_replicas|kubelet_server_expiration_renew_errors|kube_pod_container_resource_limits|container_network_transmit_bytes_total|container_network_receive_packets_dropped_total|container_memory_working_set_bytes|kube_pod_container_status_waiting_reason|container_network_receive_packets_total|kube_namespace_created|namespace_workload_pod|kube_pod_container_resource_requests|kubelet_running_pod_count|namespace_memory:kube_pod_container_resource_limits:sum|kube_deployment_status_replicas_updated|kube_statefulset_status_observed_generation|kube_deployment_status_observed_generation|container_cpu_cfs_periods_total|node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile|kubelet_certificate_manager_client_expiration_renew_errors|cluster:namespace:pod_memory:active:kube_pod_container_resource_limits|kube_daemonset_updated_number_scheduled|kubelet_volume_stats_inodes|kube_node_info|kube_deployment_spec_replicas|container_memory_rss|namespace_memory:kube_pod_container_resource_requests:sum|kubelet_volume_stats_available_bytes
                action: keep
          relabel_configs:
              - action: keep
                regex: ksm-kube-state-metrics
                source_labels:
                  - __meta_kubernetes_service_name

You can see that it's pretty much the same scheme as the Prometheus configuration has. So if you need to adjust some metrics - check the official Prometheus documentation.

Now you can just apply the configuration:

kubectl apply -f config.yaml

Grafana cloud agent deployment

When we have the configuration in place, we can proceed to the last bit - Deployment with the Grafana agent. Again, everything's described in the official documentation but this is what you need to do:

MANIFEST_URL=https://raw.githubusercontent.com/grafana/agent/main/production/kubernetes/agent-bare.yaml \
NAMESPACE=grafana-agent \
/bin/sh -c "$(curl -fsSL https://raw.githubusercontent.com/grafana/agent/release/production/kubernetes/install-bare.sh)" > deploy.yaml

and apply the manifest:

kubectl apply -f deploy.yaml

As the result we have 2 pods running in the grafana-agent namespace.

NAME                                      READY   STATUS    RESTARTS   AGE
grafana-agent-6f8b68fd6-hbnkr             1/1     Running   0          6d16h
ksm-kube-state-metrics-7d8f59c464-tb9st   1/1     Running   0          10d

Result

And guess what, now we have metrics stored in the Prometheus! If you're wondering why I communicate with localhost:9090 - check the previous chapter. This is AWS SSM port-forwarding functionality that is really useful for some basic debugging.

Wrap

So now we have the complete monitoring stack. But do we really need to use some strange port-forwarding to get there? Not really. The golden standard for the viewing of metrics is the Grafana and this is what we're gonna do in the next chapter. Stay tuned!

Part I: EC2 with Prometheus

Stepan Vrany — Tue, 22 Feb 2022 09:09:14 +0000

I'm a big fan of lean Kubernetes clusters. This means that I'm quite hesitant to run there some resource-intensive workload that can't be easily scaled. So what to do with Prometheus that fits such description?

Well, there's a pretty new feature called remote write receiver that can help us to run Prometheus somewhere else and keep Kubernetes clusters small.

So let's just quickly go through the basic setup with single Prometheus server running in standalone EC2 instance.

This tutorial is very AWS and Terraform specific but it does not necessarily means that you can't use it anywhere else.

Prometheus installation: summary

Let's summarize what we need in order to get this running:

Security group with ingress rule for 9090 port
S3 bucket for configuration files
IAM role
IAM permissions
Instance profile with role attached
EBS volume
some cloud-init that puts all the pieces together and start the Prometheus
internal hostname

Security group

So the story here is that some agent running inside the cluster will be accessing the Prometheus server. Such cluster has some Security Group attached so we can use it as the reference in our rules - var.source_security_group_id

resource "aws_security_group" "this" {
  name   = "ec2_prometheus_${var.environment}"
  vpc_id = var.vpc_id
}

resource "aws_security_group_rule" "egress_all" {
  type     = "egress"
  to_port  = 0
  protocol = "-1"
  cidr_blocks = [
    "0.0.0.0/0",
  ]
  from_port         = 0
  security_group_id = aws_security_group.this.id
}

resource "aws_security_group_rule" "ingress_prometheus" {
  type                     = "ingress"
  to_port                  = 9090
  protocol                 = "tcp"
  source_security_group_id = var.source_security_group_id
  from_port                = 9090
  security_group_id        = aws_security_group.this.id
}

Note the 0.0.0.0/0 egress rule. This is kinda classic rule that basically allow all outgoing communication. In our case we'll need it to pull Prometheus artifacts from GitHub. This might differ based on your configuration and security requirements.

S3 bucket for the configuration

S3 bucket is used as the store for prometheus config, rules config and the alertmanager config. We can also use userdata but it's no so convenient since every change in configuration files would replace the EC2 instance.

Hence we render configuration files, put them into s3 bucket and Prometheus instance will be pulling them as needed.

So the following snippet create one S3 bucket with 3 objects - config, rules and alertmanager config.

locals {
  prometheus_config = yamlencode({
    global = {
      scrape_interval = "1m",
    },
    rule_files = [
      "/etc/prometheus/prometheus.rules.yaml",
    ],
    alerting = {
      alertmanagers : [
        {
          static_configs = [
            {
              targets = [
                "localhost:9093",
              ]
            }
          ],
        }
      ],
    },
  })
}

resource "aws_s3_bucket" "config" {
  bucket = "${var.project_name}-${var.environment}"
}

resource "aws_s3_object" "prometheus_config" {
  bucket = aws_s3_bucket.config.id
  key    = "prometheus.yaml"
  content = templatefile(
    "${path.module}/prometheus.config.yaml",
    {
      environment = var.environment,
      s3_bucket   = aws_s3_bucket.config.id,
      config      = local.prometheus_config,
    }
  )

  force_destroy = true
}

resource "aws_s3_object" "prometheus_rules" {
  bucket = aws_s3_bucket.config.id
  key    = "prometheus.rules.yaml"
  content = templatefile(
    "${path.module}/prometheus.rules.config.yaml",
    {
      environment = var.environment,
      s3_bucket   = aws_s3_bucket.config.id,
      config      = var.config_prometheus_rules,
    }
  )
  force_destroy = true
}

resource "aws_s3_object" "alertmanager_config" {
  bucket = aws_s3_bucket.config.id
  key    = "alertmanager.yaml"
  content = templatefile(
    "${path.module}/alertmanager.config.yaml",
    {
      environment = var.environment,
      s3_bucket   = aws_s3_bucket.config.id,
      config      = var.config_alertmanager,
    }
  )

  force_destroy = true
}

Now let's check all the templates referenced there. The first one is the prometheus config.

# prometheus configuration
# environment: ${environment}
# bucket: ${s3_bucket}
---
${config}%

It does not contain anything special and same applies for the other files. This is the file for the rules

# prometheus configuration
# environment: ${environment}
# bucket: ${s3_bucket}
---
${config}

and this is the alertmanager configuration:

# alertmanager configuration
# environment: ${environment}
# bucket: ${s3_bucket}
---
${config}

The reason why we use just config placeholder is simple: we'd like to compose all the configuration with HCL, not YAML. Use local.prometheus_config for the reference. That's the magic.

IAM role, policy and instance profile

Everyone's playing some role. This EC2 instance is not different. It needs to communicate with s3 bucket and SSM (for the remote management) so we take this into account.

resource "aws_iam_role" "this" {
  name = "ec2_prometheus_${var.environment}"
  assume_role_policy = jsonencode({
    Statement = [{
      Action = "sts:AssumeRole"
      Effect = "Allow"
      Principal = {
        Service = "ec2.amazonaws.com"
      }
    }]
    Version = "2012-10-17"
  })
}

resource "aws_iam_policy" "this" {
  name = "ec2_prometheus_${var.environment}"

  policy = jsonencode({
    Statement = [
      # SSM Session manager stuff
      {
        Effect = "Allow"
        Action = [
          "ssm:DescribeAssociation",
          "ssm:GetDeployablePatchSnapshotForInstance",
          "ssm:GetDocument",
          "ssm:DescribeDocument",
          "ssm:GetManifest",
          "ssm:ListAssociations",
          "ssm:ListInstanceAssociations",
          "ssm:PutInventory",
          "ssm:PutComplianceItems",
          "ssm:PutConfigurePackageResult",
          "ssm:UpdateAssociationStatus",
          "ssm:UpdateInstanceAssociationStatus",
          "ssm:UpdateInstanceInformation"
        ]
        Resource = "*"
      },
      {
        Effect = "Allow"
        Action = [
          "ssmmessages:CreateControlChannel",
          "ssmmessages:CreateDataChannel",
          "ssmmessages:OpenControlChannel",
          "ssmmessages:OpenDataChannel"
        ]
        Resource = "*"
      },
      {
        Effect = "Allow"
        Action = [
          "ec2messages:AcknowledgeMessage",
          "ec2messages:DeleteMessage",
          "ec2messages:FailMessage",
          "ec2messages:GetEndpoint",
          "ec2messages:GetMessages",
          "ec2messages:SendReply"
        ]
        Resource = "*"
      },
      // s3 for the configuration
      {
        Effect = "Allow"
        Action = [
          "s3:GetBucketLocation",
          "s3:ListAllMyBuckets",
        ]
        Resource = "*"
      },
      {
        Effect = "Allow"
        Action = [
          "s3:List*",
        ]
        Resource = [
          aws_s3_bucket.config.arn,
        ]
      },
      {
        Effect = "Allow"
        Action = [
          "s3:*",
        ]
        Resource = [
          "${aws_s3_bucket.config.arn}/*",
        ]
      },
    ]
    Version = "2012-10-17"
  })
}

resource "aws_iam_role_policy_attachment" "this" {
  policy_arn = aws_iam_policy.this.arn
  role       = aws_iam_role.this.name
}

resource "aws_iam_instance_profile" "this" {
  name = "ec2_prometheus_${var.environment}"
  role = aws_iam_role.this.name
}

EC2 instance

Now we can put all the pieces together and create the EC2 instance itself. In the following snippet you can also see the EBS volume and its attachment to the instance.

resource "aws_ebs_volume" "prometheus_0" {
  availability_zone = data.aws_subnet.this.availability_zone
  size              = var.volume_size
  type              = "gp2"
}

resource "aws_instance" "prometheus_0" {
  ami = var.ami_id
  vpc_security_group_ids = [
    aws_security_group.this.id,
  ]
  subnet_id            = var.subnet_id
  iam_instance_profile = aws_iam_instance_profile.this.name
  instance_type        = var.instance_type
  tags = {
    Name = "${var.environment}_prometheus_0"
  }
  user_data_base64 = base64encode(
    templatefile(
      "${path.module}/cloud.init.yaml",
      {
        s3_bucket   = aws_s3_bucket.config.id,
        environment = var.environment,
      }
    )
  )
}

resource "aws_volume_attachment" "prometheus_0" {
  instance_id = aws_instance.prometheus_0.id
  volume_id   = aws_ebs_volume.prometheus_0.id
  device_name = "/dev/sdf"
}

Please note the user_data_base64 property in the EC2 instance definition. This section refers to the YAML template with the following content:

#cloud-config

# environment: ${environment}
runcmd:
  # install AWS CLI, neeeded for downloading of configuration files
  - |
      apt-get update && apt-get install unzip -y
      curl -Lo awscli.zip https://awscli.amazonaws.com/awscli-exe-linux-aarch64.zip
      unzip awscli.zip
      ./aws/install
      rm awscli.zip

  # install prometheus binary
  - |
      curl -Lo prometheus.tar.gz https://github.com/prometheus/prometheus/releases/download/v2.33.1/prometheus-2.33.1.linux-arm64.tar.gz
      tar -xvf prometheus.tar.gz
      cp ./prometheus-2.33.1.linux-arm64/prometheus /usr/local/bin/prometheus
      rm -rf ./prometheus-2.33.1.linux-arm64
      rm -rf prometheus.tar.gz

  # install alertmanager binary
  - |
      curl -Lo alertmanager.tar.gz https://github.com/prometheus/alertmanager/releases/download/v0.23.0/alertmanager-0.23.0.linux-arm64.tar.gz
      tar -xvf alertmanager.tar.gz
      mv ./alertmanager-0.23.0.linux-arm64/alertmanager /usr/local/bin/alertmanager
      rm -rf alertmanager-0.23.0.linux-arm64
      rm alertmanager.tar.gz

  # vait for EBS volume
  - |
      while [ ! -b $(readlink -f /dev/nvme1n1) ];
      do
        echo "waiting for device /dev/nvme1n1"
        sleep 5
      done

      # format volume
      blkid $(readlink -f /dev/nvme1n1) || mkfs -t ext4 $(readlink -f /dev/nvme1n1)

      # create a mount
      mkdir -p /data
      if ! grep "/dev/nvme1n1" /etc/fstab;
      then
        echo "/dev/nvme1n1 /data ext4 defaults,discard 0 0" >> /etc/fstab
      fi

      # mount volume
      mount /data

  # enable and start systemd services
  - |
      systemctl daemon-reload
      systemctl enable prepare-prometheus.service && systemctl start prepare-prometheus.service && sleep 10
      systemctl enable prometheus.service && systemctl start prometheus.service
      systemctl enable alertmanager.service && systemctl start alertmanager.service

write_files:

  - path: /usr/local/bin/prepare-prometheus
    permissions: '0744'
    content: |
      #!/bin/sh

      mkdir -p /etc/prometheus
      aws s3 cp s3://${s3_bucket}/prometheus.yaml /etc/prometheus/prometheus.yaml
      aws s3 cp s3://${s3_bucket}/alertmanager.yaml /etc/prometheus/alertmanager.yaml
      aws s3 cp s3://${s3_bucket}/prometheus.rules.yaml /etc/prometheus/prometheus.rules.yaml
      curl -X POST http://localhost:9090/-/reload || true

  - path: /etc/systemd/system/prepare-prometheus.service
    content: |
      [Unit]
      Description=Prepare prometheus / alertmanager configuration
      Wants=network-online.target
      After=network-online.target

      [Service]
      Type=oneshot
      ExecStart=/usr/local/bin/prepare-prometheus

  # please note data.mount in dependencies
  - path: /etc/systemd/system/prometheus.service
    content: |
      [Unit]
      Description=Prometheus
      Wants=network-online.target
      After=network-online.target data.mount prepare-prometheus.service

      [Service]
      Type=simple
      ExecStart=/usr/local/bin/prometheus \
          --config.file /etc/prometheus/prometheus.yaml \
          --storage.tsdb.path /data/ \
          --web.enable-lifecycle \
          --web.console.templates=/etc/prometheus/consoles \
          --web.console.libraries=/etc/prometheus/console_libraries \
          --enable-feature=remote-write-receiver

      [Install]
      WantedBy=multi-user.target

  - path: /etc/systemd/system/alertmanager.service
    content: |
      [Unit]
      Description=Alert Manager
      Wants=network-online.target
      After=network-online.target data.mount prepare-prometheus.service

      [Service]
      Type=simple
      ExecStart=/usr/local/bin/alertmanager \
          --config.file /etc/prometheus/alertmanager.yaml \
          --storage.path=/data/

      [Install]
      WantedBy=multi-user.target

When you start the instance with such user data, it will download all the tools required and create systemd units for prometheus and alertmanager.

Last but not least, there's also prepare-prometheus service. This oneshot (run for completion) downloads configuration files and puts them to the respective path - /etc/prometheus. And since prometheus and alertmanager services have this helper as a dependency - it's gonna happer before the prometheus starts.

Internal hostname

We'll also need to assign some hostname to the EC2 instance. The reason is obvious - we don't want to edit all the upcoming configuration files every time we replace the instance with the new one.

resource "aws_route53_zone" "prometheus" {
  name = "prometheus.local"
  vpc {
    vpc_id = var.vpc_id
  }
}

resource "aws_route53_record" "prometheus_0" {
  name = "p01.prometheus.local"
  type = "A"
  records = [
    aws_instance.prometheus_0.private_ip,
  ]
  zone_id = aws_route53_zone.prometheus.zone_id
  ttl     = 60
}

Please note the vpc block in aws_route53_zone.prometheus resource. This means that prometheus.local names will be resolvable in the VPC. We're gonna use this functionality in the upcoming chapters.

Connecting to the EC2 instance

INSTANCE_ID=$(aws ec2 describe-instances \
    --filters 'Name=tag:Name,Values=dev-ap-south-1_prometheus_0' 'Name=instance-state-name,Values=running' \
    --output text --query 'Reservations[*].Instances[*].InstanceId')

Forwarding Prometheus to localhost

aws ssm start-session --target "${INSTANCE_ID}" --document-name AWS-StartPortForwardingSession --parameters '{"portNumber":["9090"],"localPortNumber":["9090"]}'

Wrap

In this part we've gone through the basic configuration of standalone prometheus instance. Such setup does not require any manual configuration and it will handle even AMI updates - all the persistent data are stored on EBS.

In the next chapter I'm gonna show you the other side - agents running in the Kubernetes cluster.

ESP32 with Arduino CLI

Stepan Vrany — Sun, 02 May 2021 08:06:00 +0000

A few days ago I've bought this beautiful ESP32-based development board. I selected Olimex since it was the only manufacturer with PoE boards available in my region (west Europe).

Here you can see how it looks with PoE-enabled Ethernet port connected.

// Detect dark theme var iframe = document.getElementById('tweet-1388518748957691907-699'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1388518748957691907&theme=dark" }

And since I'm pretty new in this area I had to fight some battles to make it work with Arduino-cli. Here's some brief description of steps I had to do to successfully upload a sketch to the ESP32-POE-ISO.

Please note that I'm Mac OS user so some details are relevant only for Apple users.

Create a new sketch



arduino-cli sketch new 01
cd 01

Also, download some sample sketches from Olimex's GitHub account.

Arduino config file arduino-cli.yaml



board_manager:
  additional_urls:
    - https://dl.espressif.com/dl/package_esp32_index.json

Download the index



arduino-cli core update-index --config-file arduino-cli.yaml

Install the ESP32 core



arduino-cli core install esp32:esp32

List the boards



arduino-cli board list

sample output:



Port                            Type        Board Name FQBN Core

/dev/cu.Bluetooth-Incoming-Port Serial Port Unknown


/dev/cu.debug-console           Serial Port Unknown


/dev/cu.usbmodem009NTNHF15512   Serial Port Unknown


/dev/cu.usbserial-1310          Serial Port Unknown


/dev/cu.wlan-debug              Serial Port Unknown

Try to compile the sketch



arduino-cli compile --fqbn esp32:esp32:esp32-poe-iso .

Try to upload the sketch



arduino-cli upload -p /dev/cu.usbserial-1310 --fqbn esp32:esp32:esp32-poe-iso .

Monitor the serial output



screen /dev/cu.usbserial-1310 115200

Result

After 30 - 40 minutes I was able to start with some basic development. Board is now connected to the Ethernet and it obtained an IPv4 address from my DHCP server.

// Detect dark theme var iframe = document.getElementById('tweet-1388764656853000192-40'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1388764656853000192&theme=dark" }

Well, it was easier than I originally thought. Can't wait to test some more complex stuff!

Getting started with Arduino and TinyGo: MBP M1 edition

Stepan Vrany — Wed, 28 Apr 2021 07:28:31 +0000

Hey, this is gonna be a very short post. Today I wanted to play a bit with my old Arduino Uno but it took me some time to identify stuff that is specific for M1 processors.

As the first thing, we need to install all the tooling. Since there's limited support for Apple Silicons - we're gonna use Rosetta 2. Hence all the brew commands need to be prepended with arch -x86_64.

arch -x86_64 brew tap tinygo-org/tools
arch -x86_64 brew install tinygo

arch -x86_64 brew tap osx-cross/avr
arch -x86_64 brew install avr-gcc
arch -x86_64 brew install avrdude

Now we can create our first Go/Arduino project.

cd ~
mkdir app
cd app
touch main.go

And we can create the first Go sketch.

package main

import (
    "machine"
    "time"
)

func main() {
    led := machine.LED
    led.Configure(machine.PinConfig{Mode: machine.PinOutput})
    for {
        led.High()
        time.Sleep(time.Millisecond * 10000)
        led.Low()
        time.Sleep(time.Millisecond * 10000)
    }
}

This sketch will be blinking with integrated LED (ouptut 13). It's time to flash it to the Arduino!

tinygo flash -target arduino -port /dev/cu.usbmodem13201

Please note that device /dev/cu.usbmodem13201
might be a bit different in your case. You can
identify the right one by listing /dev/cu.*.