DEV Community: cosckoya

What I learned today?: Caesar cipher

cosckoya — Mon, 03 May 2021 14:35:48 +0000

Here I am studying Python and Cryptography at the same time. Just for fun. Oh! lucky man.

So, yesterday I was reading about famous cryptography methods along history, what were the purposes and how they changed in the modern times. Some of them, where really basic... but you know what they say "When a puzzle is solved is easy to understand".

So I was reading about Julius Caesar and it's crypto method for personal correspondency. (Did you know about it? Because I dont.)

This method is really simple, and it's based in a substitution type. In this case every character in the aplhabet gets a number from 0 to 25: A=0, B=1, C=2... Z=25.

Now you get a number between 0 and 25, that will be named as "key" and get a message to ecnrypt: I will use number 7 as a "key" and this message: "These Romans are crazy!" (yes, it's from Asterix comics). So...

Here we go:

Key: 7
Raw: "These Romans are crazy!"
Enc: "AOLZL YVTHUZ HYL JYHGF!"

This means that T=A and Z=G. the "original" alphabet was "moved" 7 positions backwards:

Plain	A	B	C	D	E	F	G	H	I	J	K	L	M	N	O	P	Q	R	S	T	U	V	W	X	Y	Z
Cipher	T	U	V	W	X	Y	Z	A	B	C	D	E	F	G	H	I	J	K	L	M	N	O	P	Q	R	S

With this script you can test this to encrypt:

#!/usr/bin/env python3

message = "Klaatu Barada Nitko."
key = 7
LETTERS = "ABCDEFGHIJKLMNOPQRSTUVWXYZ"

mode = "encrypt"  # set to 'encrypt' or 'decrypt'
translated = ""
message = message.upper()
# run the encryption/decryption code on each symbol in the message string
for symbol in message:
    if symbol in LETTERS:
        num = LETTERS.find(symbol)  # get the number of the symbol
        if mode == "encrypt":
            num = num + key
        elif mode == "decrypt":
            num = num - key

        if num >= len(LETTERS):
            num = num - len(LETTERS)
        elif num < 0:
            num = num + len(LETTERS)
        translated = translated + LETTERS[num]
    else:
        translated = translated + symbol

print(translated)

And this othe one to decrypt:

#!/usr/bin/env python3

message = "RSHHAB IHYHKH UPARV!"
LETTERS = "ABCDEFGHIJKLMNOPQRSTUVWXYZ"

for key in range(len(LETTERS)):
    translated = ""
    for symbol in message:
        if symbol in LETTERS:
            num = LETTERS.find(symbol)
            num = num - key
            if num < 0:
                num = num + len(LETTERS)
            translated = translated + LETTERS[num]
        else:
            translated = translated + symbol

    print("Key #%s: %s" % (key, translated))

You can find these scripts and more useful info in this book:

Title: Hacking Secret Ciphers with Python
Author: Al Sweigart
ISBN: 978-1482614374

Have fun!

Prometheus: Python metrics (with Docker and Gitlab CI)

cosckoya — Mon, 26 Apr 2021 23:08:19 +0000

Prometheus is a metric based monitoring platform also it's one of all-time my favorite tools ever. And from a time ago I was thinking to develop some project with it. What's my plan? I want to create my multi-language application cluster with Prometheus monitoring, and then add some Grafana Loki, Cortex and Thanos integrations.

What's the first step?. Integrate some Prometheus metrics library with a basic python app. To reach this, I just use the Prometheus "client_python" sample, and make a container with it and push into a public registry. So...

First step! Create a Gitlab repository

Second step! Create an "app" folder and copy-paste this code into a "main.py":

from prometheus_client import start_http_server, Summary
import random
import time

# Create a metric to track time spent and requests made.
REQUEST_TIME = Summary('request_processing_seconds', 'Time spent processing request')

# Decorate function with metric.
@REQUEST_TIME.time()
def process_request(t):
    """A dummy function that takes some time."""
    time.sleep(t)

if __name__ == '__main__':
    # Start up the server to expose the metrics.
    start_http_server(8000)
    # Generate some requests.
    while True:
        process_request(random.random())

Add a "requisites.txt" file with this content:

prometheus_client

Add a "Dockerfile" like this one:

FROM python:3.9-alpine

WORKDIR /app
COPY . .
RUN pip install -r requisites.txt

RUN chmod u+x main.py

ENTRYPOINT ["/app/main.py"]

Third Step! Create a Gitlab CI with Container Registry pipeline. To manage this task, I created a ".gitlab-ci.yaml" file in the repository basepath:

stages:
- build

image: docker:stable

services:
- docker:dind

build:
  stage: build
  when: on_success
  only:
  - master
  script:
  - docker login -u $CI_REGISTRY_USER -p $CI_JOB_TOKEN $CI_REGISTRY
  - docker build -f app/Dockerfile -t $CI_REGISTRY_IMAGE app
  - docker push $CI_REGISTRY_IMAGE

This repository should look like this:

├── app
│   ├── Dockerfile
│   ├── main.py
│   └── requisites.txt
├── .gitlab-ci.yml
└── README.md

Now, let commit all the files and wait until the pipeline is finished.

Four and last step! Run that image and scrape some metrics:

Run docker container as a daemon, with "python-prom" name, listening on 8000/TCP and delete that container when the image stops:

$> docker run -d --rm --name python-prom -p 8000:8000 registry.gitlab.com/cosckoya/python-prom

Check that the image is up & running (also listening on that port)

$> docker ps

CONTAINER ID   IMAGE                                      COMMAND          CREATED              STATUS              PORTS                                       NAMES
380cc1a00a8b   registry.gitlab.com/cosckoya/python-prom   "/app/main.py"   About a minute ago   Up About a minute   0.0.0.0:8000->8000/tcp, :::8000->8000/tcp   python-prom

And scrape those metrics!

$> curl localhost:8000

# HELP python_gc_objects_collected_total Objects collected during gc
# TYPE python_gc_objects_collected_total counter
python_gc_objects_collected_total{generation="0"} 309.0
python_gc_objects_collected_total{generation="1"} 43.0
python_gc_objects_collected_total{generation="2"} 0.0
# HELP python_gc_objects_uncollectable_total Uncollectable object found during GC
# TYPE python_gc_objects_uncollectable_total counter
python_gc_objects_uncollectable_total{generation="0"} 0.0
python_gc_objects_uncollectable_total{generation="1"} 0.0
python_gc_objects_uncollectable_total{generation="2"} 0.0
# HELP python_gc_collections_total Number of times this generation was collected
# TYPE python_gc_collections_total counter
python_gc_collections_total{generation="0"} 36.0
python_gc_collections_total{generation="1"} 3.0
python_gc_collections_total{generation="2"} 0.0
[..]

Our Python Prometheus base image is done.

On my next post, I will create a Go or Java base prometheus application and all three of these base images will be deployed into a Kubernetes cluster alongside a Prometheus and create some Prometheus workbench with them.

Running my first Gitlab CI/CD pipeline

cosckoya — Sun, 18 Apr 2021 17:28:26 +0000

I was testing some Gitlab functionalities on my last jobs, but I've never played it by myself so, I decided to migrate my personal wiki/toolbox into Gitlab pages and see how it goes.

Why Gitlab? Because I am feeling more comfortable with it. Github is a very nice SCM but I wanted to start working with Gitlab and I though that this could be a very nice opportunity to do so.

So, I migrated my repository from Github to Gitlab, just pushing all the files to an empty Gitlab repository project. That's all.

Also I created a ".gitlab-ci.yaml" file with a few steps to control the deployment:

image: python:3.8-buster

before_script:
  - pip install -r requirements.txt

test:
  stage: test
  script:
  - mkdocs build --strict --verbose --site-dir test
  artifacts:
    paths:
    - test
  except:
  - master

pages:
  stage: deploy
  script:
  - mkdocs build --strict --verbose --site-dir public
  artifacts:
    paths:
    - public
  only:
  - master

Leaving this file in the repository root path, now the project looks like this:

CMD> tree -a -L 1                       
.
├── docs
├── .git
├── .gitignore
├── .gitlab-ci.yml
├── mkdocs.yml
├── README.md
└── requirements.txt

2 directories, 5 files

When I just committed my changes into the repository, everything starts to run and the CI finally just deployed my project into Gitlab Pages. Here is the result: Cosckoya Gitlab pages.

Amazing.

I really want to test some more Gitlab CI/CD (Terraform, Docker, Helm) and integrations (Hangouts, Slack) with this new toy of mine.

Enjoy!

Terraform v15.0 with AWS (EKS deployment)

cosckoya — Sat, 17 Apr 2021 22:14:50 +0000

Terraform v15 was released on April 14th.

On this post I will use the following resources:

· Provision an EKS Cluster (AWS)
· Terraform v15.0
· Terraform Registry
· Pre-Commit
· Terraform Pre-commit
· Terraform-docs
· Tflint
· Tfsec

This is based on the Hashicorp Learning Tutorial "Provision an EKS Cluster (AWS)". I took that tutorial as a base project for this post, then I had tweaked it a little bit. I had set up some variables, modified some module and providers versions and changed some of the "list" and "map" functions. Also, I added some pre-commit hooks based on tflint, tfsec and terraform-docs. Let's start!

Before starting, I always make some resource inventory/manifest, more like a shopping cart, that helps me on what should I deploy for each cloud project. So here is the "inventory":

1 x Amazon VPC
6 x Amazon Subnet (3 x Public + 3 x Private)
3 x Amazon EC2
1 x Amazon EKS
1 x Kubernetes AWS-Auth policy

First thing to do, is to init a git repository. Mine is this one. Then we must create some folder and files there:

CMD> mkdir terraform
CMD> touch terraform/{main,outputs,variables,versions}.tf terraform/README.md

To enable pre-commit, we need to create a .pre-commit-config.yaml file that will contain relative config and install pre-commit in the project:

CMD> echo 'repos:
- repo: git://github.com/antonbabenko/pre-commit-terraform
  rev: master
  hooks:
  - id: terraform_fmt
  - id: terraform_validate
  - id: terraform_docs
  - id: terraform_docs_without_aggregate_type_defaults
  - id: terraform_tflint
    args:
    - 'args=--enable-rule=terraform_documented_variables'
  - id: terraform_tfsec
- repo: https://github.com/pre-commit/pre-commit-hooks
  rev: master
  hooks:
  - id: check-merge-conflict
  - id: end-of-file-fixer' > .pre-commit-config.yaml

CMD> pre-commit install
pre-commit installed at .git/hooks/pre-commit

Project folder hierarchy should look like this:

CMD> tree 
.
├── README.md
└── terraform
    ├── main.tf
    ├── outputs.tf
    ├── README.md
    ├── variables.tf
    └── versions.tf

Let's start setting up "versions.tf" file. This file contains our relative provider versions, it's better to close this versions to avoid some misconfigurations in the future updates:

# Providers version
# Ref. https://www.terraform.io/docs/configuration/providers.html

terraform {
  required_version = "~>0.15"
  required_providers {
    # Base Providers
    random = {
      source  = "hashicorp/random"
      version = "3.1.0"
    }
    null = {
      source  = "hashicorp/null"
      version = "3.1.0"
    }
    local = {
      source = "hashicorp/local"
      version = "2.1.0"
    }
    template = {
      source = "hashicorp/template"
      version = "2.2.0"
    }
    # AWS Provider
    aws = {
      source  = "hashicorp/aws"
      version = "3.37.0"
    }
    # Kubernetes Provider
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = "2.1.0"
    }    
  }
}

Then I manage to set some project variables in the "variables.tf". This will help me to make some different workspaces, environments or command line variable override:

# Common
variable "project" {
  default     = "cosckoya"
  description = "Project name"
}

variable "environment" {
  default     = "laboratory"
  description = "Environment name"
}

# Amazon
variable "region" {
  default     = "us-east-1"
  description = "AWS region"
}

variable "vpc_cidr" {
  type        = string
  default     = "10.0.0.0/16"
  description = "AWS VPC CIDR"
}

variable "public_subnets_cidr" {
  type        = list(any)
  default     = ["10.0.1.0/24", "10.0.2.0/24", "10.0.3.0/24"]
  description = "AWS Public Subnets"
}

variable "private_subnets_cidr" {
  type        = list(any)
  default     = ["10.0.4.0/24", "10.0.5.0/24", "10.0.6.0/24"]
  description = "AWS Private Subnets"
}

This is my custom "main.tf" file. Here I changed the "locals" to set some tags as a "tomap(..)" function, update d the modules to the last version and also updated the Kubernetes version to 1.19. Just to test and have fun.

[...]
locals {
  cluster_name = "${var.project}-${var.environment}-eks"
  tags = tomap({"Environment" = var.environment, "project" = var.project})
}
[...]
## Amazon Networking
module "vpc" {
  source = "terraform-aws-modules/vpc/aws"
  version = "2.78.0"
  name                 = "${var.project}-${var.environment}-vpc"
  cidr                 = var.vpc_cidr
  azs                  = data.aws_availability_zones.available.names
  private_subnets      = var.public_subnets_cidr
  public_subnets       = var.private_subnets_cidr
  enable_nat_gateway   = true
  single_nat_gateway   = true
  enable_dns_hostnames = true

  tags = {
    "kubernetes.io/cluster/${local.cluster_name}" = "shared"
  }

  public_subnet_tags = {
    "kubernetes.io/cluster/${local.cluster_name}" = "shared"
    "kubernetes.io/role/elb"                      = "1"
  }

  private_subnet_tags = {
    "kubernetes.io/cluster/${local.cluster_name}" = "shared"
    "kubernetes.io/role/internal-elb"             = "1"
  }
}
[...]

I didn't touch any of the "outputs.tf".

Check these git repositories for more information:
· Learn Terraform - Provision an EKS Cluster
· Cosckoya's AWS Terraform Laboratory

Time to have fun now. Let's play with this:

· Initialize the project

CMD> terraform init

In here we should test the pre-commit rules that we had set up, take note of every Tfsec error about the security compliance. Try to resolve each or comment them with this docs

Add these lines to the "terraform" README.md:

<!-- BEGINNING OF PRE-COMMIT-TERRAFORM DOCS HOOK -->
<!-- END OF PRE-COMMIT-TERRAFORM DOCS HOOK -->

Running Pre-Commit, our config will create a basic document with a lot of interesting information about our Terraform project

CMD> pre-commit run -a
Terraform fmt............................................................Passed
Terraform validate.......................................................Passed
Terraform docs...........................................................Passed
Terraform docs (without aggregate type defaults).........................Passed
Terraform validate with tflint...........................................Passed
Check for merge conflicts................................................Passed
Fix End of Files.........................................................Passed

And check that README.md file:

CMD> cat terraform/README.md
[...]
## Requirements
[...]
## Providers
[...]
## Modules
[...]
## Resources
[...]
## Inputs
[...]
## Outputs

Mine is available here

Now it's time to have fun!
· Plan the project

CMD> terraform plan

· Deploy the project

CMD> terraform apply

· Connect to the cluster and enjoy!

CMD> aws eks --region $(terraform output -raw region) update-kubeconfig --name $(terraform output -raw cluster_name)

Running some basic commands we can see that the cluster is up and running:

CMD> kubectl cluster-info

Kubernetes control plane is running at https://<SOME-BIG-HASH>.us-east-1.eks.amazonaws.com
CoreDNS is running at https://<SOME-BIG-HASH>.us-east-1.eks.amazonaws.com/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

CMD> kubectl get nodes

NAME                         STATUS   ROLES    AGE   VERSION
ip-10-0-2-138.ec2.internal   Ready    <none>   26m   v1.19.6-eks-d1db3c
ip-10-0-2-88.ec2.internal    Ready    <none>   26m   v1.19.6-eks-d1db3c
ip-10-0-3-68.ec2.internal    Ready    <none>   26m   v1.19.6-eks-d1db3c

Enjoy!

Ps. As you could see this is so similar to the AWS Terraform Learn page. Little tweaks to test some changes between versions.

I'm a very big fan of @antonbabenko work. I recommend everyone to follow him.

Kali on a container

cosckoya — Fri, 02 Apr 2021 15:20:48 +0000

If you ask me about Kali some years ago, I must be thinking about Temple of Doom and if you asked me about hacking I would think about that blind man in Sneakers film :P

But joking apart, what can I say to you about Kali?

Kali is a cooked linux release that allows anyone to test pentesting tools. Could be installed as a virtual machine (VirtualBox, Vagrant...), as a boot partition on your computer or be deployed as EC2/VM instance on AWS/GCP/Azure/whatever cloud. These options are OK but it takes a "lot" of time to run, tweak and maintain a Kali system.

Imagine that you want to try a Kali tool and you dont want to install Kali Desktop, there is another option: Kali Docker images.

Kali images contains a base image without tools but all them APT repositories are enabled. So it's really easy to install and test anything on them.

Let's try to run some web scanner with Nikto, DMitry and SSLScan.

First you need to run bash in the Kali docker image:

docker run -ti --rm kalilinux/kali-rolling bash

Now, in this prompted shell we could run as many commands as we need:

┌──(root💀616f2bee7ea0)-[/]
└─#

Let's start running some APT commands to install our tools:

┌──(root💀616f2bee7ea0)-[/]
└─# apt update && apt install nikto sslscan dmitry -y

That's it. Tools are been installed. Let's check that are ready to run:

Nikto

┌──(root💀616f2bee7ea0)-[/]
└─# nikto -Version                                  
[...]
File                               Version      Last Mod
-----------------------------      --------     ----------
Nikto main                         2.1.6        
[...]

Dmitry

┌──(root💀616f2bee7ea0)-[/]
└─# dmitry -version
Deepmagic Information Gathering Tool
"There be some deep magic going on"

Version: DMitry/1.3a (Unix)

SSL Scan

┌──(root💀616f2bee7ea0)-[/]
└─# sslscan --version
                2.0.9-static
                OpenSSL 1.1.1l-dev  xx XXX xxxx

Imagine all the possibilities:

Create your custom Kali images:

FROM kalilinux/kali-rolling

RUN apt-get update -qq \
 && apt-get install -qq -y --no-install-recommends \
    nikto sslscan dmitry metasploit-framework
[...]

Run a Kali image into a Kubernetes cluster:

kubectl run prompt-shell --generator=run-pod/v1 --rm -i --tty --image kalilinux/kali -- bash

There are a lot of possibilities here.

Time to have fun. Enjoy!

Reference

DockerHub

Kind: Tiniest Kubernetes ever!

cosckoya — Fri, 02 Apr 2021 13:27:04 +0000

Kind is the smallest, tiniest and funniest Kubernetes runtime I've tried.

Kind could be used for different purposes: to run a local environment, include as a stage on a CD/CD pipeline, test new Kubernetes releases for your OCI compliance images.

Kind requires Docker properly installed on your computer, otherwise it wont deploy anything.

Let's start creating a cluster!

First things to do is install Kind. There are a few ways to install it: Install Go language then run "go get" command or more easy to go, download latest release from the Github repository :D

I already automated this "download" task from my dotfiles, here is the "code":

mkdir ${HOME}/bin ;\
wget -q https://github.com/kubernetes-sigs/kind/releases/latest/download/kind-linux-amd64 -O ${HOME}/bin/kind ;\
chmod u+x ${HOME}/bin/kind ;\
${HOME}/bin/kind version

This will prompt you a message like this:

kind v0.10.0 go1.15.7 linux/amd64

Next step is to create a cluster, to acknoledge this we must run this command:

CMD> kind create cluster --name tiny-k8s-01

So, we now have a running Kind K8s cluster!

There a lot of funny features that makes Kind a versatile K8s tool, like:

Run as many K8s clusters as you want:

for i in {0..4}; do
kind create cluster --name tiny-cluster-0$i
done

Cluster list:

CMD> kind get clusters

tiny-cluster-00
tiny-cluster-01
tiny-cluster-02
tiny-cluster-03
tiny-cluster-04

Create Kind cluster with different Kubernetes releases:

CMD> kind create cluster --image kindest/node:v1.19.7 --name tiny-cluster-v19

Kind images are aviable in this DockerHub registry

There are a lot more features to be used with Kind like Ingress, build images and import into Kind, check your OCI rootless ...

Enjoy!

References:

InfluxDB v2 on Kubernetes

cosckoya — Sun, 28 Mar 2021 22:26:10 +0000

Here is the thing, let's setup a InfluxDB v2 in Kubernetes. Fast and simple and just with 3 YAML files ;)

You will need a Kubernetes cluster. You could create a GKE, EKS or AKS, but if you're poor as I am; use Minikube :P

Install Minikube or Kind and start it:

Minikube:

CMD> minikube start

Kind:

CMD> kind create cluster

Then create a namespace called "monitoring"

CMD> kubectl create namespace monitoring

Easy, dont you think? What now? Let's create a Service Account with a privileged permission scheme, this is mine:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: cthulhu
  namespace: monitoring
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: cthulhu-clusterrole
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: cthulhu
  namespace: monitoring
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: cthulhu-role
  namespace: monitoring
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: cthulhu
  namespace: monitoring

Take note that this one is just for testing purposes, NEVER use this RBAC on a production environment, you will need to touch here and there for a production environment. Did I told you that this will be fast as a shark?

Let's deploy our InfluxDB v2 server. Here is the YAML:

apiVersion: v1
kind: Service
metadata:
  name: influxdb
  namespace: monitoring 
spec:
  type: ClusterIP
  selector:
    app: influxdb
  ports:
  - name: api
    port: 9999
  - name: gui
    port: 8086
--------
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: influxdb
  namespace: monitoring   
spec:
  serviceName: "influxdb"
  selector:
    matchLabels:
      app: influxdb
  template:
    metadata:
      labels:
        app: influxdb
    spec:
      serviceAccount: cthulhu    
      containers:
      - name: influxdb
        image: quay.io/influxdb/influxdb:v2.0.3
        resources:
          limits:
            memory: "128Mi"
            cpu: 1
        ports:
        - name: api
          containerPort: 9999
        - name: gui
          containerPort: 8086
        volumeMounts:
        - name: data
          mountPath: /root/.influxdbv2
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 1Gi
        volumeMode: Filesystem

This is a Statefulset that will be enough to test new influxDB v2 features on your local environment, develop some alerting and dashboards with the new GUI and test some API things.

Now, we must configure the basic credentials. You could manage to create the admin user via GUI, but let's do it via YAML, because could be useful to create some pipelines with this monitoring monster.

So, here is the "config" YAML. Take note that I set up a simple token and password. Did I say that this is only for testing purposes?

apiVersion: batch/v1
kind: Job
metadata:
  name: influxdb-setup
  namespace: monitoring
spec:
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: create-credentials
          image: quay.io/influxdb/influxdb:v2.0.3
          command:
            - influx
          args:
            - setup
            - --host
            - http://influxdb.monitoring:8086
            - --bucket
            - kubernetes
            - --org
            - InfluxData
            - --password
            - yogsothoth
            - --username
            - admin
            - --token
            - secret-token
            - --force

So, so... what's next? I also have a Telegraf to send some data to that Influx, here it is:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: telegraf
  namespace: monitoring  
spec:
  selector:
    matchLabels:
      app: telegraf
  template:
    metadata:
      labels:
        app: telegraf
    spec:
      serviceAccount: cthulhu
      volumes:
      - name: config
        configMap:
          name: telegraf
      containers:
      - name: telegraf
        image: telegraf:alpine
        imagePullPolicy: IfNotPresent
        volumeMounts:
        - name: config
          mountPath: /etc/telegraf/telegraf.conf
          subPath: telegraf.conf     
--------
apiVersion: v1
kind: ConfigMap
metadata:
  name: telegraf
  namespace: monitoring  
data:
  telegraf.conf: | 
    [global_tags]
      infra = "minikube"

    [agent]

      interval            = "10s"
      round_interval      = true
      metric_batch_size   = 1000
      metric_buffer_limit = 10000
      collection_jitter   = "0s"
      flush_interval      = "10s"
      flush_jitter        = "0s"
      precision           = ""
      debug               = false
      quiet               = false
      logfile             = ""
      hostname            = "telegraf"
      omit_hostname       = false

    [[outputs.influxdb_v2]]
      urls         = ["http://influxdb:8086"]

      organization = "InfluxData"
      bucket       = "kubernetes"
      token        = "secret-token"

    [[inputs.cpu]]  
      percpu           = true
      totalcpu         = true
      collect_cpu_time = false
      report_active    = false

    [[inputs.disk]]
       ignore_fs = ["rootfs","tmpfs", "devtmpfs", "devfs", "iso9660", "overlay", "aufs", "squashfs"]

And then? Nothing more than port-forward InfluxDB GUI service and connect to http://localhost:8086

kubectl port-forward service/influxdb 8086:8086

Enjoy!

Prometheus: Alertmanager with SendGrid and Slack API

cosckoya — Sun, 28 Mar 2021 21:54:16 +0000

I've been working in with Prometheus a lot, it's about 3 years now when I changed my mind from Nagios / Zabbix monitoring to a metrics monitoring. I was developing lot of different Perl, Python and Shell scripts to monitor infrastructures and applications via SNMP, NPRE o NSCA "protocols".

I discovered Kubernetes in their second year of life, and then Prometheus. You can read a lot of different Prometheus implementations like adding a Grafana, maybe doing some bigger Prometheus with Cortex or Thanos, but what is really useful to me on my personal projects? Setting up Alertmanager.

Im here I will show you how to manage AlertManager. At first I installed it with Helm and configured a custom values.yaml file, but let's make it simple. I will show you how to edit YAMLs and later how to custom Helm charts :D

AlertManager

Configure AlertManager

AlertManager is the tool that sends all notifications via mail or API. Prometheus server has all the alert rules, when an alert is triggered by a rule, Alertmanager will send the notification.

First step is to configure Alertmanager config file (alertmanager.yml)

In here we could add some customization parameters like:

Global: Where we can set our notification sender by SMTP (SendGrid, Gmail...), API (Slack, WeChat..) or simply set the timeouts.
Route: We set a sending logic here, how the mails and to who the notifications will be sent.
Inhibit Rules: Add another set of rules to set diffferent route rules to different receivers
Receivers: Here we set the different receivers. Maybe we need to send notifications to slack, mail, or both. In here we need to add as much receviers that we want to.

In this sample I will set up an API (via Slack) and a mail (via SendGrid)

global:
  # Sendgrid SMTP properties.
  smtp_smarthost: 'smtp.sendgrid.net:587'
  smtp_from: 'Alertmanager <alertmanager@cosckoya.io>'
  smtp_auth_username: 'apikey'
  smtp_auth_password: '<API KEY HERE>'

  # Slack API properties
  slack_api_url: 'https://hooks.slack.com/services/<API TOKEN HERE>'

  receivers:
  - name: mail
    email_configs:
    - to: "admins@mail.me"
      headers:
        Subject: "Alert ({{ .Status }}): {{ .CommonLabels.severity }} {{ .CommonAnnotations.message }} ({{ .CommonLabels.alertname }})"
      html: |
        Greetings,
        <p>
        You have the following firing alerts:
        <ul>
        {{ range .Alerts }}
        <li>{{.Labels.alertname}} on {{.Labels.instance}}</li>
        <li>Labels:</li>
        <li>{{ range .Labels.SortedPairs }} - {{ .Name }} = {{ .Value }}</li>
        <li>{{ end }}Annotations:</li>
        <li>{{ range .Annotations.SortedPairs }} - {{ .Name }} = {{ .Value }}</li>
        <li>{{ end }}---</li>
        {{ end }}
        </ul>
        </p>

  - name: slack     
    slack_configs:
    - channel: '#alerting'
      send_resolved: true
      icon_url: https://avatars3.githubusercontent.com/u/3380462
      title: "|-"
       [{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}] {{ .CommonLabels.alertname }} for {{ .CommonLabels.job }}
       {{- if gt (len .CommonLabels) (len .GroupLabels) -}}
         {{" "}}(
         {{- with .CommonLabels.Remove .GroupLabels.Names }}
           {{- range $index, $label := .SortedPairs -}}
             {{ if $index }}, {{ end }}
             {{- $label.Name }}="{{ $label.Value -}}"
           {{- end }}
         {{- end -}}
         )
       {{- end }}
      text: >-
       {{ range .Alerts -}}
       *Alert:* {{ .Annotations.title }}{{ if .Labels.severity }} - `{{ .Labels.severity }}`{{ end }}

       *Description:* {{ .Annotations.description }}

       *Details:*
         {{ range .Labels.SortedPairs }} • *{{ .Name }}:* `{{ .Value }}`
         {{ end }}
       {{ end }}

  route:
    group_wait: 10s
    group_interval: 5m
    receiver: mail
    repeat_interval: 10s
    routes:
    - match:
        severity: High
      repeat_interval: 1m
      receiver: slack

This parameters will setup AlertManager to send notifications to the declared "routes"; mail as default and Slack API when the match case is setted up.

Now we have to setup some alerts. These ones are just for demostration and testing that the alert triggers works.

  groups:
  - name: deadman.rules
    rules:
    - alert: deadman
      expr: vector(1)
      for: 10s
      labels:
        severity: Critical
      annotations:
        message: Dummy Check
        summary: This is a dummy check meant to ensure that the entire lerting pipeline is functional.
  - name: pod.rules
    rules:
    - alert: PodLimit
      expr: kubelet_running_pods > 0
      for: 15m
      labels:
        severity: High
      annotations:
        message: 'Kubelet {{$labels.instance}} is running {{$value}} pods.'

Last Step is to setup "alerting" in the prometheus server config file (prometheus.yml):

[...]
  alerting:
    alertmanagers:
      - static_configs:
        - targets:
          - prometheus-alertmanager:80
[...]

If you managed to set it up correctly, then you will see a "gatling gun" Alertmanager notifications in your mail and Slack channel.

Enjoy!

ASDF or how to manage multiple runtime versions.

cosckoya — Sun, 28 Mar 2021 21:32:02 +0000

ASDF is a linux tool that allows you to manage multiple runtime versions. But.. what that means?.

Imagine that you have to work with different Helm charts and Terraform infrastructures; or maybe you want to test a new cli release for Popeye, Etcd, Github Cli o Firebase. You should download latest release try it and go back to the last one that worked fine.

With ASDF you can switch between versions whenever you want, set that version at three stages: Global, Local and Shell.

Are you interested? OK! Then you should try it. Download you release from it's official Github page: https://github.com/asdf-vm/asdf

You will need to clone the repo:

git clone https://github.com/asdf-vm/asdf.git ${HOME}/.asdf

And then add that path to your $PATH:

export PATH=$PATH:${HOME}/.asdf

Now you need to update the plugin list:

CMD> asdf plugin list all

Output sample:
[...]
kubebuilder                   https://github.com/virtualstaticvoid/asdf-kubebuilder.git
kube-capacity                 https://github.com/looztra/asdf-kube-capacity.git
kubectl                      *https://github.com/Banno/asdf-kubectl.git
kubectl-bindrole              https://github.com/looztra/asdf-kubectl-bindrole.git
kubectx                      *https://gitlab.com/wt0f/asdf-kubectx.git
kubefedctl                    https://github.com/kvokka/asdf-kubefedctl.git
kubeseal                      https://github.com/stefansedich/asdf-kubeseal.git
kubesec                       https://github.com/vitalis/asdf-kubesec.git
kubespy                       https://github.com/jfreeland/asdf-kubespy.git
kubeval                       https://github.com/stefansedich/asdf-kubeval.git
terraform                    *https://github.com/Banno/asdf-hashicorp.git
terraform-docs               *https://github.com/looztra/asdf-terraform-docs.git
terraform-lsp                 https://github.com/bartlomiejdanek/terraform-lsp.git
terraform-validator           https://github.com/looztra/asdf-terraform-validator.git
terragrunt                    https://github.com/lotia/asdf-terragrunt.git
[...]

We want to install Kubectl so let's check which versions are available:

CMD> asdf list all kubectl 

Output:
1.17.14
1.17.15
1.17.16
1.17.17
1.18.12
1.18.13
1.18.14
1.18.15
1.18.16
1.18.17
1.19.4
1.19.5
1.19.6
1.19.7
1.19.8
1.19.9
1.20.0-rc.0
1.20.0-beta.2
1.20.0
1.20.1
1.20.2
1.20.3
1.20.4
1.20.5
1.21.0-beta.0
1.21.0-rc.0
1.21.0-alpha.1
1.21.0-beta.1
1.21.0-alpha.2
1.21.0-alpha.3

Mmm... lot of them, so lets install 1.17.15 version and 1.18.13:

CMD> asdf install kubectl 1.17.15
CMD> asdf install kubectl 1.18.13

Done. Let's check that everything is installed properly:

CMD> asdf list

Output:

kubectl
  1.17.15
  1.18.13

Nice! Seems that both versions are installed properly, now let's work with them:

First, let's try and set v1.17.15:

CMD> asdf shell kubectl 1.17.15
CMD> kubectl version --client=true --short=true

Client Version: v1.17.16-rc.0

OK! Let's try v1.18.13:

CMD> asdf shell kubectl 1.18.13
CMD> kubectl version --client=true --short=true

Client Version: v1.18.13

Nice! So I managed to install both versions ans switch in the same shell session. If I would like to set for as a recurrent version, I would run "asdf local kubectl 1.1x.y" or maybe for a global environment.

This is one of the best linux tools ever. I could manage almost any runtime just in seconds. Really useful and productive!

If you want to check how I manage my tools, check my dotfile Github repository: https://github.com/cosckoya/.dotfiles

Enjoy!

Mkdocs: Setting up mkdocs.yml

cosckoya — Sun, 28 Mar 2021 14:41:31 +0000

Setting up mkdocs.yml file is needed in the way to deploy a Mkdocs project, in this file we can set a lot of different values related to the theme, add custom CSS, navigation index, social network and more other things.

Let start with basic parameters:

site_name: '<Site Name>'
site_description: '<Site Description>'
site_author: '<user>'
site_url: 'https://<user>.github.io/<git_repo>'
copyright: "copyright &copy; 202X <user>"

repo_url: 'https://github.com/<user>/<git_repo>'
edit_uri: ""

Next we could add some social networks that will be show in the Mkdocs project:

extra:
  social:
    - icon: fontawesome/brands/github
      link: https://github.com/cosckoya
    - icon: fontawesome/brands/dev
      link: https://dev.to/cosckoya

These last two parameter blocks enable some basic functionalities and behaviors. But also we could manage some advanced things about the theme.

Themes can be built-in or custom. Im using a built-in one, Material.

To add this theme, I followed the Material docs and set up the following parameters:

theme:
  name: material
  logo: img/logo.png
  favicon: img/favicon.png
  language: en
  palette:
    scheme: slate
    primary: deep purple
    accent: deep purple
  icon:
    repo: fontawesome/brands/github

Last step is to set Navigation, where I added a plugin (mkdocs-literate-nav). With this plugin I could remove "nav" settings from mkdocs.yml file, add some "SUMMARY.md" files on each directory adding this block:

plugins:
  - literate-nav:
      nav_file: SUMMARY.md

A SUMMARY.md file must exist in each folder, will be automatically detected and should have the following format:

* [Klaatu Barada Nitko!](index.md)
* [Code](code/)
* [Linux](linux/)
* [Databases](databases/)
* [Cloud](cloud/)
* [Containers](container/)
* [Security](security/)
* [DevOps](devops/)
* [Monitoring](monitoring/)
* [Bookmarks](bookmarks.md)
* [About](about.md)

That's it. Enjoy!

Mkdocs automate CI/CD with Github actions

cosckoya — Tue, 23 Mar 2021 01:03:02 +0000

I'm developing my own mkdocs as a personal wiki. I've been working as a system engineer for 10 years and I have a lot of notes, cheats, scripts and tips on my laptops. And it's been a mess to look for some of them when I really needed. That's because I started a mkdocs project :)

I've been working on it, installing all the pip packages manually, testing the builds and then publishing into github pages. One time, and another, and another the same repetitive task. So I've managed a Github action to publish the mkdocs in Github pages automatically.

But, what's Github actions? GH Actions are a feature that allows anyone to generate a CI/CD pipeline into your own git repository with no need to manage Jenkins, TravisCI or any other engine to realize that.

As you create a Github repository, you're able to create "Actions" just clicking on the project menu, then it will show you a set of different predefined actions or create a blank one. I created one from scratch:

 name: mkdocs

on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

jobs:
  build:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        python-version: [ 3.8 ]

    steps:
      - uses: actions/checkout@v2

      - name: Set up Python ${{ matrix.python-version }}
        uses: actions/setup-python@v2
        with:
          python-version: ${{ matrix.python-version }}
          architecture: x64

      - name: Install requirements
        run: python3 -m pip install -r requirements.txt

      - name: Deploy 🚀
        run: python3 -m mkdocs gh-deploy --force

This job is run when a commit or pull request is merged in the "main" branch, then it creates an Ubuntu agent with a matrix of different Python version; in here I only use Python v3.8; then it installs all the pip packages declared in the "requirements.txt" file and then runs the mkdocs command "gh-deploy" that builds and deploy the mkdocs project in the "gh-pages" branch.

Now I only have to commit my local changes into the "main" branch and Github deploys everything for me :P

Enjoy!

Create your own mkdocs with Github pages

cosckoya — Sun, 21 Mar 2021 16:40:58 +0000

Here is the beginning of a series of posts that will explain how to create your own documentation project with mkdocs. For those who doesnt know what is mkdocs. Mkdocs is a static site generator (SSG) developed in Python that allows anyone to create websites fast; source files are written in Markdown and the configuration is managed by YAML files.

First of all we need to start our mkdocs project is install Python and configure Pip.

Install mkdocs with Pip:

python -m pip install mkdocs --user

Check that mkdocs was correctly installed:

$ mkdocs --version

mkdocs, version 1.1.2 from /home/cosckoya/.local/lib/python3.8/site-packages/mkdocs (Python 3.8)

It's party time! By now mkdocs is already installed and we can create a new project with the following command:

mkdocs new my-mkdocs-project
cd my-mkdocs-project

By now you can build and check a base template project, to do that just run:

mkdocs build --clean
mkdocs serve

This commands would show the following message:

Serving on http://127.0.0.1:8000

And opening that URL will see your mkdocs project running :)

But, what happens if you want to create an online mkdocs project? You could use a lot of different static website providers like AWS S3, Google Firebase, Azure Static Website ... lot of options, the fast one for me is to publish it on a Github pages site.

To do that, we must create a new github repository like mine: https://github.com/cosckoya/wiki. In the "main" branch we will maintain the mkdocs project and in the "gh-pages" branch will be the static content of the mkdocs, this will be generated with the "mkdocs build" command.

We must initialize a repository with the base project folder:

Here is the thing. We can develop our mkdocs project: add some templates, web content, addons or images. After check that the site has everything and after some "build & serve"; we could deploy the site running:

mkdocs gh-deploy

Then navigate to your github pages site. Mine is this:

· https://cosckoya.github.io/wiki

And that's it. Enjoy!

Reference:
· https://www.mkdocs.org
· https://pages.github.com