DEV Community: Ruan Bekker

ArgoCD Getting Started - Hands On

Ruan Bekker — Sun, 05 May 2024 07:28:24 +0000

In this tutorial I will show you how to get started with ArgoCD on Kubernetes and we will cover the following topics:

How to provision a local Kubernetes cluster.
How to deploy ArgoCD on Kubernetes using Helm.
How to grant ArgoCD access to you Private Github Repositories.
How to configure your application sets on Github, and how to deploy applications to your cluster.
How to get started with RBAC to create a local user.
How to setup SSO with Authentik.
How to use Argo CD Notifications using Email.

Pre-Requisites

To follow along in this tutorial you will need the following

Kind installed (if you are provisioning Kubernetes with it)
Helm
Kubectl

Install a Kubernetes Cluster

If you already have a Kubernetes Cluster, you can skip this step.

Define the kind-config.yaml



kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  image: kindest/node:v1.26.6@sha256:6e2d8b28a5b601defe327b98bd1c2d1930b49e5d8c512e1895099e4504007adb
  extraPortMappings:
  - containerPort: 80
    hostPort: 80
    protocol: TCP
    listenAddress: "0.0.0.0"
  - containerPort: 443
    hostPort: 443
    protocol: TCP

Then create the cluster with kind:



kind create cluster --name example --config kind-config.yaml

ArgoCD Installation

You can deploy Argo CD using the kubernetes manifests and deploy them with kubectl or you can deploy them with helm.

I will be deploying Argo CD using Helm, the reason for that is, I would eventually like to manage my argo deployment using Argo CD, and I have found when deploying it initially using manifests, it was not as smooth as compared to helm.

So to start deploying Argo CD with Helm, so first we will need to add the helm chart repository where the chart is hosted:



helm repo add argo https://argoproj.github.io/argo-helm

Then we can find the latest version using the following:



helm search repo argo/argo-cd
# NAME          CHART VERSION   APP VERSION DESCRIPTION
# argo/argo-cd  6.0.13          v2.10.0     A Helm chart for Argo CD

Now since we have the version, we can get the default values and redirect the output to a file:



helm show values argo/argo-cd --version 6.0.13 > values.yaml

I only have one config parameter that I want to change and the rest I want to keep at defaults, so I am only defining this as my values.yaml:



---
configs:
  params:
    server.insecure: true

Now we can deploy argo cd to our cluster:



helm upgrade --install argocd argo/argo-cd \
  --version 6.0.13 \
  --values values.yaml \
  --namespace argocd --create-namespace

We can monitor our installation and ensure that all the pods are running in the argocd namespace:



kubectl get pods -n argocd

Once the pods are running, we can retrieve the argo cd admin password from a kubernetes secret:



kubectl get secret argocd-initial-admin-secret -n argocd \
  -o jsonpath="{.data.password}" | base64 -d

Now that we have the secret we can create a port-forward session so that we can access the argo cd frontend:



kubectl -n argocd port-forward svc/argocd-server 8080:80

Access the UI on http://localhost:8080

Once we login we should see this:

You will see a blank canvas, we do see an option to create an application via the user interface, but we are not going to use this as we will define all our resources in a declarative manner.

Continue Reading

To access the rest of this post, feel free to see the updated version on my blog:

https://ruan.dev/blog/2024/26/03/getting-started-with-argo-cd-on-kubernetes---the-ultimate-guide

How to use the MySQL Terraform Provider

Ruan Bekker — Mon, 17 Jul 2023 13:00:00 +0000

In this tutorial we will provision a MySQL Server with Docker and then use Terraform to provision MySQL Users, Database Schemas and MySQL Grants with the MySQL Terraform Provider.

About

Terraform is super powerful and can do a lot of things. And it shines when it provisions Infrastructure. So in a scenario where we use Terraform to provision RDS MySQL Database Instances, we might still want to provision extra MySQL Users, or Database Schemas and the respective MySQL Grants.

Usually you will logon to the database and create them manually with sql syntax. But in this tutorial we want to make use of Docker to provision our MySQL Server and we would like to make use of Terraform to provision the MySQL Database Schemas, Grants and Users.

Instead of using AWS RDS, I will be provisioning a MySQL Server on Docker so that we can keep the costs free, for those who are following along.

We will also go through the steps on how to rotate the database password that we will be provisioning for our user.

MySQL Server

First we will provision a MySQL Server on Docker Containers, I have a docker-compose.yaml which is available in my quick-starts github repository:

version: "3.8"

services:
  mysql:
    image: mysql:8.0
    container_name: mysql
    ports:
      - 3306:3306
    environment:
      - MYSQL_DATABASE=sample
      - MYSQL_ROOT_PASSWORD=rootpassword

Once you have saved that in your current working directory, you can start the container with docker compose:

docker-compose up -d

You can test the mysql container by logging onto the mysql server with the correct auth:

docker exec -it mysql mysql -u root -prootpassword -e 'show databases;'

This should be more or less the output:

+--------------------+
| Database           |
+--------------------+
| information_schema |
| mysql              |
| performance_schema |
| sample             |
| sys                |
+--------------------+

Terraform

If you don't have Terraform installed, you can install it from their documentation.

If you want the source code of this example, its available in my terraform-mysql/petoju-provider repository. Which you can clone and jump into the terraform/mysql/petoju-provider directory.

First we will define the providers.tf:

terraform {
  required_providers {
    mysql = {
      source = "petoju/mysql"
      version = "3.0.37"
    }
  }
}

provider "mysql" {
  alias    = "local"
  endpoint = "127.0.0.1:3306"
  username = "root"
  password = "rootpassword"
}

Then the main.tf:

resource "random_password" "user_password" {
  length           = 24
  special          = true
  min_special      = 2
  override_special = "!#$%^&*()-_=+[]{}<>:?"
  keepers = {
    password_version = var.password_version
  }
}

resource "mysql_database" "user_db" {
  provider = mysql.local
  name = var.database_name
}

resource "mysql_user" "user_id" {
  provider = mysql.local
  user = var.database_username
  plaintext_password = random_password.user_password.result
  host = "%"
  tls_option = "NONE"
}

resource "mysql_grant" "user_id" {
  provider = mysql.local
  user = var.database_username
  host = "%"
  database = var.database_name
  privileges = ["SELECT", "UPDATE"]
  depends_on = [
    mysql_user.user_id
  ]
}

Then the variables.tf:

variable "database_name" {
  description = "The name of the database that you want created."
  type        = string
  default     = null
}

variable "database_username" {
  description = "The name of the database username that you want created."
  type        = string
  default     = null
}

variable "password_version" {
  description = "The password rotates when this value gets updated."
  type        = number
  default     = 0
}

Then our outputs.tf:

output "user" {
  value = mysql_user.user_id.user
}

output "password" {
  sensitive = true
  value = random_password.user_password.result
}

Our terraform.tfvars that defines the values of our variables:

database_name     = "foobar"
database_username = "ruanb"
password_version  = 0

Now we are ready to run our terraform code, which will ultimately create a database, user and grants. Outputs the encrypted string of your password which was encrypted with your keybase_username.

Initialise Terraform:

terraform init

Run the plan to see what terraform wants to provision:

terraform plan

And we can see the following resources will be created:

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # mysql_database.user_db will be created
  + resource "mysql_database" "user_db" {
      + default_character_set = "utf8mb4"
      + default_collation     = "utf8mb4_general_ci"
      + id                    = (known after apply)
      + name                  = "foobar"
    }

  # mysql_grant.user_id will be created
  + resource "mysql_grant" "user_id" {
      + database   = "foobar"
      + grant      = false
      + host       = "%"
      + id         = (known after apply)
      + privileges = [
          + "SELECT",
          + "UPDATE",
        ]
      + table      = "*"
      + tls_option = "NONE"
      + user       = "ruanb"
    }

  # mysql_user.user_id will be created
  + resource "mysql_user" "user_id" {
      + host               = "%"
      + id                 = (known after apply)
      + plaintext_password = (sensitive value)
      + tls_option         = "NONE"
      + user               = "ruanb"
    }

  # random_password.user_password will be created
  + resource "random_password" "user_password" {
      + bcrypt_hash      = (sensitive value)
      + id               = (known after apply)
      + keepers          = {
          + "password_version" = "0"
        }
      + length           = 24
      + lower            = true
      + min_lower        = 0
      + min_numeric      = 0
      + min_special      = 2
      + min_upper        = 0
      + number           = true
      + numeric          = true
      + override_special = "!#$%^&*()-_=+[]{}<>:?"
      + result           = (sensitive value)
      + special          = true
      + upper            = true
    }

Plan: 4 to add, 0 to change, 0 to destroy.

Changes to Outputs:
  + password = (sensitive value)
  + user     = "ruanb"

Run the apply which will create the database, the user, sets the password and applies the grants:

terraform apply

Then our returned output should show something like this:

Apply complete! Resources: 4 added, 0 changed, 0 destroyed.

Outputs:

password = <sensitive>
user = "ruanb"

As our password is set as sensitive, we can access the value with terraform output -raw password, let's assign the password to a variable:

DBPASS=$(terraform output -raw password)

Then we can exec into the mysql container and logon to the mysql server with our new credentials:

docker exec -it mysql mysql -u ruanb -p$DBPASS

And we can see we are logged onto the mysql server:

Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 14
Server version: 8.0.33 MySQL Community Server - GPL

mysql>

If we run show databases; we should see the following:

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| foobar             |
| information_schema |
| performance_schema |
+--------------------+
3 rows in set (0.03 sec)

If we want to rotate the mysql password for the user, we can update the password_version variable either in our terraform.tfvars or via the cli. Let's pass the variable in the cli and do a terraform plan to verify the changes:

terraform plan -var password_version=1

And due to our value for the random resource keepers parameter being updated, it will trigger the value of our password to be changed, and that will let terraform update our mysql user's password:

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  ~ update in-place
-/+ destroy and then create replacement

Terraform will perform the following actions:

  # mysql_user.user_id will be updated in-place
  ~ resource "mysql_user" "user_id" {
        id                 = "ruanb@%"
      ~ plaintext_password = (sensitive value)
        # (5 unchanged attributes hidden)
    }

  # random_password.user_password must be replaced
-/+ resource "random_password" "user_password" {
      ~ bcrypt_hash      = (sensitive value)
      ~ id               = "none" -> (known after apply)
      ~ keepers          = { # forces replacement
          ~ "password_version" = "0" -> "1"
        }
      ~ result           = (sensitive value)
        # (11 unchanged attributes hidden)
    }

Plan: 1 to add, 1 to change, 1 to destroy.

Let's go ahead by updating our password:

terraform apply -var password_version=1 -auto-approve

To validate that the password has changed, we can try to logon to mysql by using the password variable that was created initially:

docker exec -it mysql mysql -u ruanb -p$DBPASS

And as you can see authentication failed:

mysql: [Warning] Using a password on the command line interface can be insecure.
ERROR 1045 (28000): Access denied for user 'ruanb'@'localhost' (using password: YES)

Set the new password to the variable again:

DBPASS=$(terraform output -raw password)

Then try to logon again:

docker exec -it mysql mysql -u ruanb -p$DBPASS

And we can see we are logged on again:

Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 22
Server version: 8.0.33 MySQL Community Server - GPL

mysql>

Resources

The terraform mysql provider:

https://registry.terraform.io/providers/petoju/mysql/latest/docs

The quick-starts repository:

https://github.com/ruanbekker/quick-starts

Thank You

Thanks for reading, feel free to check out my website, feel free to subscribe to my newsletter or follow me at @ruanbekker on Twitter.

Linktree: https://go.ruan.dev/links
Patreon: https://go.ruan.dev/patreon

How to use the AWS Terraform Provider

Ruan Bekker — Sun, 16 Jul 2023 00:27:33 +0000

In this post we will be using the AWS Terraform provider, from how to install Terraform, create a AWS IAM User, configure the AWS Provider and deploy a EC2 instance using Terraform.

AWS IAM User

In order to authenticate against AWS’s APIs, we need to create a AWS IAM User and create Access Keys for Terraform to use to authenticate.

From https://aws.amazon.com/ logon to your account, then search for IAM:

Select IAM, then select “Users” on the left hand side and select “Create User”, then provide the username for your AWS IAM User:

Now we need to assign permissions to our new AWS IAM User. For this scenario I will be assigning a IAM Policy directly to the user and I will be selecting the “AdministratorAccess” policy. Keep in mind that this allows admin access to your whole AWS account:

Once you select the policy, select “Next” and select “Create User”. Once the user has been created, select “Users” on the left hand side, search for your user that we created, in my case “medium-terraform”.

Select the user and click on “Security credentials”. If you scroll down to the “Access keys” section, you will notice we don’t have any access keys for this user:

In order to allow Terraform access to our AWS Account, we need to create access keys that Terraform will use, and because we assigned full admin access to the user, Terraform will be able to manage resources in our AWS Account.

Click “Create access key”, then select the “CLI” option and select the confirmation at the bottom:

Select “Next” and then select “Create access key”. I am providing a screenshot of the Access Key and Secret Access Key that has been provided, but by the time this post has been published, the key will be deleted.

Store your Access Key and Secret Access Key in a secure place and treat this like your passwords. If someone gets access to these keys they can manage your whole AWS Account.

I will be using the AWS CLI to configure my Access Key and Secret Access Key, as I will configure Terraform later to read my Access Keys from the Credential Provider config.

First we need to configure the AWS CLI by passing the profile name, which I have chosen medium for this demonstration:

aws --profile medium configure

We will be asked to provide the access key, secret access key, aws region and the default output:

AWS Access Key ID [None]: AKIATPRT2G4SGXLAC3HJ
AWS Secret Access Key [None]: KODnR[............]nYTYbd
Default region name [None]: eu-west-1
Default output format [None]: json

To verify if everything works as expected we can use the following command to verify:

aws --profile medium sts get-caller-identity

The response should look something similar to the following:

{
    "UserId": "AIDATPRT2G4SOAO5Y7S5Z",
    "Account": "000000000000",
    "Arn": "arn:aws:iam::000000000000:user/medium-terraform"
}

Terraform

Now that we have our AWS IAM User configured, we can install Terraform, if you don’t have Terraform installed yet, you can follow their Installation Documentation.

Once you have Terraform installed, we can setup our workspace where we will ultimately deploy a EC2 instance, but before we get there we need to create our project directory and change to that directory:

mkdir ~/terraform-demo
cd ~/terraform-demo

Then we will create 4 files with .tf extensions:

touch main.tf
touch outputs.tf
touch providers.tf
touch variables.tf

We will define our Terraform definitions on how we want our desired infrastructure to look like. We will get to the content in the files soon.

I personally love Terraform’s documentation as they are rich in examples and really easy to use.

Head over to the Terraform AWS Provider documentation and you scroll a bit down, you can see the Authentication and Configuration section where they outline the order in how Terraform will look for credentials and we will be making use of the shared credentials file as that is where our access key and secret access key is stored.

If you look at the top right corner of the Terraform AWS Provider documentation, they show you how to use the AWS Provider:

We can copy that code snippet and paste it into our providers.tf file and configure the aws provider section with the medium profile that we’ve created earlier.

This will tell Terraform where to look for credentials in order to authenticate with AWS.

Open providers.tf with your editor of choice:

terraform {
  required_providers {
    aws = {
      source = "hashicorp/aws"
      version = "5.8.0"
    }
  }
}

provider "aws" {
  shared_credentials_files = ["~/.aws/credentials"]
  profile                  = "medium"
  region                   = "eu-west-1"
}

Then we can open main.tf and populate the following to define the EC2 instance that we want to provision:

data "aws_ami" "latest_ubuntu" {
  most_recent = true
  owners = ["099720109477"]

  filter {
    name   = "name"
    values = ["ubuntu/images/hvm-ssd/ubuntu-jammy-22.04-*-server-*"]
  }

  filter {
    name   = "architecture"
    values = ["x86_64"]
  }
}

resource "aws_instance" "ec2" {
  ami           = data.aws_ami.latest_ubuntu.id
  instance_type = var.instance_type
  tags = {
    Name = "${var.instance_name}-ec2-instance"
  }
}

In the above example we are filtering for the latest Ubuntu 22.04 64bit AMI then we are defining a EC2 instance and specifying the AMI ID that we filtered from our data source.

Note that we haven’t specified a SSH Keypair, as we are just focusing on how to provision a EC2 instance.

As you can see we are also referencing variables, which we need to define in variables.tf :

variable "instance_name" {
  description = "Instance Name for EC2."
  type        = string
  default     = "test"
}

variable "instance_type" {
  description = "Instance Type for EC2."
  type        = string
  default     = "t2.micro"
}

And then lastly we need to define our outputs.tf which will be used to output the instance id and ip address:

output "instance_id" {
  value = aws_instance.ec2.id
}

output "ip" {
  value = aws_instance.ec2.public_ip
}

Deploy our EC2 with Terraform

Now that our infrastructure has been defined as code, we can first initialise terraform which will initialise the backend and download all the providers that has been defined:

terraform init

Once that has done we can run a “plan” which will show us what Terraform will deploy:

terraform plan

Now terraform will show us the difference in what we have defined, and what is actually in AWS, as we know its a new account with zero infrastructure, the diff should show us that it needs to create a EC2 instance.

The response from the terraform plan shows us the following:

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # aws_instance.ec2 will be created
  + resource "aws_instance" "ec2" {
      + ami                                  = "ami-0f56955469757e5aa"
      + arn                                  = (known after apply)
      + id                                   = (known after apply)
      + instance_type                        = "t2.micro"
      + key_name                             = (known after apply)
      + private_ip                           = (known after apply)
      + public_ip                            = (known after apply)
      + security_groups                      = (known after apply)
      + subnet_id                            = (known after apply)
      + tags                                 = {
          + "Name" = "test-ec2-instance"
        }
      + tags_all                             = {
          + "Name" = "test-ec2-instance"
        }
      + vpc_security_group_ids               = (known after apply)
    }

Plan: 1 to add, 0 to change, 0 to destroy.

Changes to Outputs:
  + instance_id = (known after apply)
  + ip          = (known after apply)

As you can see terraform has looked up the AMI ID using the data source, and we can see that terraform will provision 1 resource which is a EC2 instance. Once we hare happy with the plan, we can run a apply which will show us the same but this time prompt us if we want to proceed:

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

aws_instance.ec2: Creating...
aws_instance.ec2: Still creating... [10s elapsed]
aws_instance.ec2: Still creating... [20s elapsed]
aws_instance.ec2: Still creating... [30s elapsed]
aws_instance.ec2: Creation complete after 35s [id=i-005c08b899229fff0]

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

Outputs:

instance_id = "i-005c08b899229fff0"
ip = "34.253.196.167"

And now we can see our EC2 instance was provisioned and our outputs returned the instance id as well as the public ip address.

We can also confirm this by looking at the AWS EC2 Console:

Note that Terraform Configuration is idempotent, so when we run a terraform apply again, terraform will check what we have defined as what we want our desired infrastructure to be like, and what we actually have in our AWS Account, and since we haven’t made any changes there should be no changes.

We can run a terraform apply to validate that:

terraform apply

And we can see the response shows:

data.aws_vpc.selected: Reading...
data.aws_ami.latest_ubuntu: Reading...
data.aws_ami.latest_ubuntu: Read complete after 1s [id=ami-0f56955469757e5aa]
data.aws_vpc.selected: Read complete after 1s [id=vpc-063d7ac3124053dfa]
data.aws_subnet.selected: Reading...
data.aws_subnet.selected: Read complete after 1s [id=subnet-0b7acd7593611c1bb]
aws_instance.ec2: Refreshing state... [id=i-005c08b899229fff0]

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

Cleanup

Destroy the infrastructure that we provisioned:

terraform destroy

It will show us what terraform will destroy, then upon confirming we should see the following output:

Plan: 0 to add, 0 to change, 1 to destroy.

Changes to Outputs:
  - instance_id = "i-005c08b899229fff0" -> null
  - ip          = "34.253.196.167" -> null

Do you really want to destroy all resources?
  Terraform will destroy all your managed infrastructure, as shown above.
  There is no undo. Only 'yes' will be accepted to confirm.

  Enter a value: yes

aws_instance.ec2: Destroying... [id=i-005c08b899229fff0]
aws_instance.ec2: Still destroying... [id=i-005c08b899229fff0, 10s elapsed]
aws_instance.ec2: Still destroying... [id=i-005c08b899229fff0, 20s elapsed]
aws_instance.ec2: Still destroying... [id=i-005c08b899229fff0, 30s elapsed]
aws_instance.ec2: Destruction complete after 31s

Destroy complete! Resources: 1 destroyed.

If you followed along and you also want to clean up the AWS IAM user, head over to the AWS IAM Console and delete the “medium-terraform” IAM User.

Thank You

I hope you enjoyed this post, I will be posting more terraform related content.

Should you want to reach out to me, you can follow me on Twitter at @ruanbekker or check out my website at https://ruan.dev

How to Run a AMD64 Bit Linux VM on a Mac M1

Ruan Bekker — Fri, 26 May 2023 21:16:08 +0000

This tutorial will show you how you can run 64bit Ubuntu Linux Virtual Machines on a Apple Mac M1 arm64 architecture macbook using UTM.

Installation

Head over to their documentation and download the UTM.dmg file and install it, once it is installed and you have opened UTM, you should see this screen:

Creating a Virtual Machine

In my case I would like to run a Ubuntu VM, so head over to the Ubuntu Server Download page and download the version of choice, I will be downloading Ubuntu Server 22.04, once you have your ISO image downloaded, you can head over to the next step which is to "Create a New Virtual Machine":

I will select "Emulate" as I want to run a amd64 bit architecture, then select "Linux":

In the next step we want to select the Ubuntu ISO image that we downloaded, which we want to use to boot our VM from:

Browse and select the image that you downloaded, once you selected it, it should show something like this:

Select continue, then select the architecture to x86_64, the system I kept on defaults and the memory I have set to 2048MB and cores to 2 but that is just my preference:

The next screen is to configure storage, as this is for testing I am setting mine to 8GB:

The next screen is shared directories, this is purely optional, I have created a directory for this:



mkdir ~/utm

Which I've then defined for a shared directory, but this depends if you need to have shared directories from your local workstation.

The next screen is a summary of your choices and you can name your vm here:

Once you are happy select save, and you should see something like this:

You can then select the play button to start your VM.

The console should appear and you can select install or try this vm:

This will start the installation process of a Linux Server:

Here you can select the options that you would like, I would just recommend to ensure that you select Install OpenSSH Server so that you can connect to your VM via SSH.

Once you get to this screen:

The installation process is busy and you will have to wait a couple of minutes for it to complete. Once you see the following screen the installation is complete:

On the right hand side select the circle, then select CD/DVD and select the ubuntu iso and select eject:

Starting your VM

Then power off the guest and power on again, then you should get a console login, then you can proceed to login, and view the ip address:

SSH to your VM

Now from your terminal you should be able to ssh to the VM:

We can also verify that we are running a 64bit vm, by running uname --processor:

Thank You

Thanks for reading, feel free to check out my website, feel free to subscribe to my newsletter or follow me at @ruanbekker on Twitter.

Linktree: https://go.ruan.dev/links
Patreon: https://go.ruan.dev/patreon

Dasel - jq for yaml json and toml

Ruan Bekker — Sat, 15 Apr 2023 23:32:05 +0000

I stumbled upon a great tool called Dasel which enables you to query and modify data structures using selector strings.

I was working on a Rust project and wanted a way to query Toml files, similar to how you would use jq for json data.

Dasel was exactly what I was looking for.

Installation

If you are on Mac you can use homebrew to install:

brew install dasel

If you are on Linux you can install Dasel using:

wget https://github.com/TomWright/dasel/releases/download/v2.1.2/dasel_linux_amd64
install -o root -g root -m 0755 dasel_linux_amd64 /usr/bin/dasel

YAML

For the first demonstration we will use the following deployment.yaml file:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - image: nginx:1.15.0
        name: nginx
        ports:
        - containerPort: 80

Let's say we wanted to query the number of replicas:

$ dasel -f deployment.yaml -r yaml '.spec.replicas'                                                     
2

If we wanted to view the image:

$ dasel -f deployment.yaml -r yaml '.spec.template.spec.containers.[0].image'
nginx:1.14.2

We can use put to query and write information back to the yaml file. In this example we will update the container image:

dasel put -t string -v nginx:1.15.0 -f deployment.yaml -r yaml '.spec.template.spec.containers.[0].image'

To view if the image was updated:

dasel -f deployment.yaml -r yaml '.spec.template.spec.containers.[0].image'                           
nginx:1.15.0

TOML

For toml configuration, lets say the file we are working with is Cargo.toml:

[package]
name = "helloworld"
version = "0.1.0"
edition = "2021"

To view the name:

$ dasel -f Cargo.toml -r toml '.package.name'
helloworld

JSON

For JSON data, we can query it like this:

$ echo '{"names": [{"name":{"first":"Tom","last":"Wright"}}]}' | dasel -r json 'names.[0].name.first'
Tom

We can also use the filter function:

$ echo '{"names": [{"name":"Tom","last":"Wright"},{"name":"John","last":"Wrong"}]}' | dasel -r json 'names.all().filter(name)'

Thank You

Thanks for reading.

Create a Discord Bot in Python

Ruan Bekker — Mon, 13 Jun 2022 06:55:36 +0000

In this tutorial we will develop our own Discord bot using Python.

The source code for this bot will be stored in my github repository

About the bot

First we will create a basic discord bot that will greet the message sender, and then we will create a Minecraft Bot, that will enable us to do the following:

:: Bot Usage ::
!mc help : shows help
!mc serverusage : shows system load in percentage
!mc serverstatus : shows if the server is online or offline
!mc whoisonline : shows who is online at the moment

Let's get into it.

Dependencies

Create a python virtual environment and install the dependent packages:

$ python3 -m virtualenv .venv
$ source .venv/bin/activate
$ pip install discord
$ pip install python-dotenv

Create the Discord Application

We first need to create the application on discord and retrieve a token that our python app will require.

Create a application on discord:

https://discordapp.com/developers/applications

You should see:

Click "New Application" and provide it a name:

Checkout our latest product - the ultimate tailwindcss page creator 🚀

Once you create the application you will get a screen to upload a logo, provide a description and most importantly get your application id as well as your public key:

Then select the Bot section:

Then select "Add Bot":

Select OAuth2 and select the "bot" scope:

At the bottom of the page it will provide you with a URL that looks something like:

https://discord.com/api/oauth2/authorize?client_id=xxxxxxxxxxx&permissions=0&scope=bot

Paste the link in your browser and authorize the bot to your server of choice:

Then click authorize, and you should see your bot appearing on Discord:

Developing the Discord Bot

Now we will be building our python discord bot, head back to the "Bot" section and select "Reset Token", then copy and store the token value to a file .env:

DISCORD_TOKEN=xxxxxxxxx

So in our current working directory, we should have a file .env with the following content:

$ cat .env
DISCORD_TOKEN=your-unique-token-value-will-be-here

For this demonstration, I will create a private channel in discord called minecraft-test and add the bot MinecraftBot to the channel (this is only for testing, after testing you can add your bot to your other channels for other people to use):

For our first test, a basic bot, where we would like to type hello and the bot should greet us by our username, in our mc_discord_bot.py file we will have:

import discord
import os
from dotenv import load_dotenv

BOT_NAME = "MinecraftBot"

load_dotenv()
DISCORD_TOKEN = os.getenv("DISCORD_TOKEN")

bot = discord.Client()

@bot.event
async def on_ready():
    print(f'{bot.user} has logged in.')

@bot.event
async def on_message(message):
    if message.author == bot.user:
        return
    if message.content == 'hello':
        await message.channel.send(f'Hey {message.author}')
    if message.content == 'goodbye':
        await message.channel.send(f'Goodbye {message.author}')

bot.run(DISCORD_TOKEN)

Then run the bot:

$ python mc_discord_bot.py
MinecraftBot has logged in.

And when we type hello and goodbye you can see our bot responds on those values:

Now that we tested our bot, we can clear the mc_discord_bot.py and write our minecraft bot, the requirements of this bot is simple, but we would like the following:

use the command !mc to trigger our bot and subcommands for what we want
able to see who is playing minecraft on our server at the moment
able to get the status if the minecraft server is online
able to get the server load percentage (as the bot runs on the minecraft server)

This is our complete mc_discord_bot.py:

import discord
from discord.ext import commands
import requests
import os
from dotenv import load_dotenv
import random
import multiprocessing

# Variables
BOT_NAME = "MinecraftBot"
load_dotenv()
DISCORD_TOKEN = os.getenv("DISCORD_TOKEN")

minecraft_server_url = "lightmc.fun" # this is just an example, and you should use your own minecraft server

bot_help_message = """
:: Bot Usage ::
`!mc help` : shows help
`!mc serverusage` : shows system load in percentage
`!mc serverstatus` : shows if the server is online or offline
`!mc whoisonline` : shows who is online at the moment
"""

available_commands = ['help', 'serverusage', 'serverstatus', 'whoisonline']

# Set the bot command prefix
bot = commands.Bot(command_prefix="!")

# Executes when the bot is ready
@bot.event
async def on_ready():
    print(f'{bot.user} succesfully logged in!')

# Executes whenever there is an incoming message event
@bot.event
async def on_message(message):
    print(f'Guild: {message.guild.name}, User: {message.author}, Message: {message.content}')
    if message.author == bot.user:
        return

    if message.content == '!mc':
        await message.channel.send(bot_help_message)

    if 'whosonline' in message.content:
        print(f'{message.author} used {message.content}')
    await bot.process_commands(message)

# Executes when the command mc is used and we trigger specific functions
# when specific arguments are caught in our if statements
@bot.command()
async def mc(ctx, arg):
    if arg == 'help':
        await ctx.send(bot_help_message)

    if arg == 'serverusage':
        cpu_count = multiprocessing.cpu_count()
        one, five, fifteen = os.getloadavg()
        load_percentage = int(five / cpu_count * 100)
        await ctx.send(f'Server load is at {load_percentage}%')

    if arg == 'serverstatus':
        response = requests.get(f'https://api.mcsrvstat.us/2/{minecraft_server_url}').json()
        server_status = response['online']
        if server_status == True:
            server_status = 'online'
        await ctx.send(f'Server is {server_status}')

    if arg == 'whoisonline':
        response = requests.get('https://api.mcsrvstat.us/2/{minecraft_server_url}').json()
        players_status = response['players']
        if players_status['online'] == 0:
            players_online_message = 'No one is online'
        if players_status['online'] == 1:
            players_online_username = players_status['list'][0]
            players_online_message = f'1 player is online: {players_online_username}'
        if players_status['online'] > 1:
            po = players_status['online']
            players_online_usernames = players_status['list']
            joined_usernames = ", ".join(players_online_usernames)
            players_online_message = f'{po} players are online: {joined_usernames}'
        await ctx.send(f'{players_online_message}')

bot.run(DISCORD_TOKEN)

And now we can start our bot:

$ python mc_discord_bot.py

And we can run our help command:

!mc help

Which will prompt our help message, and then test out the others:

Resources

Thank you to the following authors, which really helped me doing this:

Thank You

Thanks for reading, if you like my content, check out my website, read my newsletter or follow me at @ruanbekker on Twitter.

The source code for this bot will be stored in my github repository:

github.com/ruanbekker

I've started a brand new Discord server, not much happening at the moment, but planning to share and distribute tech content and a place for like minded people to hang out. If that's something you are interested in, feel free to join on this link

Setup a Self-Hosted Git Service with Gitea

Ruan Bekker — Fri, 22 Oct 2021 09:18:20 +0000

Gitea is a self-hosted git service written in Go, it's super lightweight to run and supports ARM architectures as well, so you can run it on a Raspberry Pi as well.

What are we doing today

In this tutorial we will be setting up a self hosted version control repository with Gitea on Docker using Traefik as our Load Balancer and SSL terminations for LetsEncrypt certificates.

We will then create a example git repository, add our ssh key to our account and clone our repository using ssh, change some code, commit and push to our repository.

Assumptions

I will assume that you have docker and docker-compose installed. If you need more info on Traefik you can have a look at their website, but I have also written a post on setting up Traefik v2 in detail, but we will touch on that in this post.

Environment Details

I have 1 DNS entry set to the following:

Traefik: traefik.rbkr.xyz
Gitea: git.rbkr.xyz

Accessing our service will be done over HTTPS on port 443, and for cloning over SSH, the port will be set to 222

Directory Structure

Create the gitea directory which will be our docker compose project directory:



mkdir gitea
cd gitea

Create the traefik directory for acme.json where certificate data will be stored, create the file and change permissions on the file:



touch traefik/acme.json
chmod 600 traefik/acme.json

Traefik

Open the docker-compose.yml and add the first bit which will Traefik, ensure that you replace the following:

me@example.com with your email under certificatesResolvers.letsencrypt.acme.email
traefik.rbkr.xyz with your fqdn for traefik under traefik.http.routers.api.rule=Host()

The section for traefik:



version: '3.8'

services:
  gitea-traefik:
    image: traefik:2.4
    container_name: gitea-traefik
    restart: unless-stopped
    volumes:
      - ./traefik/acme.json:/acme.json
      - /var/run/docker.sock:/var/run/docker.sock
    networks:
      - public
    labels:
      - 'traefik.enable=true'
      - 'traefik.http.routers.api.rule=Host(`traefik.rbkr.xyz`)'
      - 'traefik.http.routers.api.entrypoints=https'
      - 'traefik.http.routers.api.service=api@internal'
      - 'traefik.http.routers.api.tls=true'
      - 'traefik.http.routers.api.tls.certresolver=letsencrypt'
    ports:
      - 80:80
      - 443:443
    command:
      - '--api'
      - '--providers.docker=true'
      - '--providers.docker.exposedByDefault=false'
      - '--entrypoints.http=true'
      - '--entrypoints.http.address=:80'
      - '--entrypoints.http.http.redirections.entrypoint.to=https'
      - '--entrypoints.http.http.redirections.entrypoint.scheme=https'
      - '--entrypoints.https=true'
      - '--entrypoints.https.address=:443'
      - '--certificatesResolvers.letsencrypt.acme.email=me@example.com`'
      - '--certificatesResolvers.letsencrypt.acme.storage=acme.json'
      - '--certificatesResolvers.letsencrypt.acme.httpChallenge.entryPoint=http'
      - '--log=true'
      - '--log.level=INFO'
    logging:
      driver: "json-file"
      options:
        max-size: "1m"

networks:
  public:
    name: public
```

We can start traefik so long:

```bash
docker-compose up -d
```

## Gitea

Now we will add the Gitea components, I have opted in for the gitea service and a redis cache and will be making use of sqlite as this is just a demonstration. For non-test environments, you can have a look at MySQL or Postres:
- https://docs.gitea.io/en-us/install-with-docker/#databases

Review the following configuration options:

- `DOMAIN` and `SSH_DOMAIN` (this will be used in your clone urls)
- `ROOT_URL` (this is set to use the HTTPS protocol, including my domain)
- `SSH_LISTEN_PORT` (this is the port listening for SSH inside the container)
- `SSH_PORT` (this is the port we are exposing from outside, which will be replaced in the clone url)
- `DB_TYPE` (Im using sqlite for this example)
- `traefik.http.routers.gitea.rule=Host()` (the host header to reach gitea via web)
- `./data/gitea` (I am persisting the data in my local working directory under the given path)

The gitea portion of the `docker-compose.yml`:

```yaml
---
version: '3.8'

services:
  ...
  gitea:
    container_name: gitea
    image: gitea/gitea:${GITEA_VERSION:-1.14.5}
    restart: unless-stopped
    depends_on:
      gitea-traefik:
        condition: service_started
      gitea-cache:
        condition: service_healthy
    environment:
      - APP_NAME="Gitea"
      - USER_UID=1000
      - USER_GID=1000
      - USER=git
      - RUN_MODE=prod
      - DOMAIN=git.rbkr.xyz
      - SSH_DOMAIN=git.rbkr.xyz
      - HTTP_PORT=3000
      - ROOT_URL=https://git.rbkr.xyz
      - SSH_PORT=222
      - SSH_LISTEN_PORT=22
      - DB_TYPE=sqlite3
      - GITEA__cache__ENABLED=true
      - GITEA__cache__ADAPTER=redis
      - GITEA__cache__HOST=redis://gitea-cache:6379/0?pool_size=100&idle_timeout=180s
      - GITEA__cache__ITEM_TTL=24h
    ports:
      - "222:22"
    networks:
      - public
    volumes:
      - ./data/gitea:/data
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.gitea.rule=Host(`git.rbkr.xyz`)"
      - "traefik.http.routers.gitea.entrypoints=https"
      - "traefik.http.routers.gitea.tls.certresolver=letsencrypt"
      - "traefik.http.routers.gitea.service=gitea-service"
      - "traefik.http.services.gitea-service.loadbalancer.server.port=3000"
    logging:
      driver: "json-file"
      options:
        max-size: "1m"

  gitea-cache:
    container_name: gitea-cache
    image: redis:6-alpine
    restart: unless-stopped
    networks:
      - public
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 15s
      timeout: 3s
      retries: 30
    logging:
      driver: "json-file"
      options:
        max-size: "1m"
...

```

So my complete `docker-compose.yml` will look like the following:

```yaml
---
version: '3.8'

services:
  gitea-traefik:
    image: traefik:2.4
    container_name: gitea-traefik
    restart: unless-stopped
    volumes:
      - ./traefik/acme.json:/acme.json
      - /var/run/docker.sock:/var/run/docker.sock
    networks:
      - public
    labels:
      - 'traefik.enable=true'
      - 'traefik.http.routers.api.rule=Host(`traefik.rbkr.xyz`)'
      - 'traefik.http.routers.api.entrypoints=https'
      - 'traefik.http.routers.api.service=api@internal'
      - 'traefik.http.routers.api.tls=true'
      - 'traefik.http.routers.api.tls.certresolver=letsencrypt'
    ports:
      - 80:80
      - 443:443
    command:
      - '--api'
      - '--providers.docker=true'
      - '--providers.docker.exposedByDefault=false'
      - '--entrypoints.http=true'
      - '--entrypoints.http.address=:80'
      - '--entrypoints.http.http.redirections.entrypoint.to=https'
      - '--entrypoints.http.http.redirections.entrypoint.scheme=https'
      - '--entrypoints.https=true'
      - '--entrypoints.https.address=:443'
      - '--certificatesResolvers.letsencrypt.acme.email=me@example.com'
      - '--certificatesResolvers.letsencrypt.acme.storage=acme.json'
      - '--certificatesResolvers.letsencrypt.acme.httpChallenge.entryPoint=http'
      - '--log=true'
      - '--log.level=INFO'
    logging:
      driver: "json-file"
      options:
        max-size: "1m"

  gitea:
    container_name: gitea
    image: gitea/gitea:${GITEA_VERSION:-1.14.5}
    restart: unless-stopped
    depends_on:
      gitea-traefik:
        condition: service_started
      gitea-cache:
        condition: service_healthy
    environment:
      - APP_NAME="Gitea"
      - USER_UID=1000
      - USER_GID=1000
      - USER=git
      - RUN_MODE=prod
      - DOMAIN=git.rbkr.xyz
      - SSH_DOMAIN=git.rbkr.xyz
      - HTTP_PORT=3000
      - ROOT_URL=https://git.rbkr.xyz
      - SSH_PORT=222
      - SSH_LISTEN_PORT=22
      - DB_TYPE=sqlite3
      - GITEA__cache__ENABLED=true
      - GITEA__cache__ADAPTER=redis
      - GITEA__cache__HOST=redis://gitea-cache:6379/0?pool_size=100&idle_timeout=180s
      - GITEA__cache__ITEM_TTL=24h
    ports:
      - "222:22"
    networks:
      - public
    volumes:
      - ./data/gitea:/data
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.gitea.rule=Host(`git.rbkr.xyz`)"
      - "traefik.http.routers.gitea.entrypoints=https"
      - "traefik.http.routers.gitea.tls.certresolver=letsencrypt"
      - "traefik.http.routers.gitea.service=gitea-service"
      - "traefik.http.services.gitea-service.loadbalancer.server.port=3000"
    logging:
      driver: "json-file"
      options:
        max-size: "1m"

  gitea-cache:
    container_name: gitea-cache
    image: redis:6-alpine
    restart: unless-stopped
    networks:
      - public
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 15s
      timeout: 3s
      retries: 30
    logging:
      driver: "json-file"
      options:
        max-size: "1m"

networks:
  public:
    name: public
```

Once your configuration is updated, start gitea:

```bash
docker-compose up -d

Creating network "public" with the default driver
Creating gitea-traefik ... done
Creating gitea-cache   ... done
Creating gitea         ... done
```

Once the containers has started, verify that they are all up:

```bash
docker-compose ps
    Name                   Command                  State                                       Ports
--------------------------------------------------------------------------------------------------------------------------------------
gitea           /usr/bin/entrypoint /bin/s ...   Up             0.0.0.0:222->22/tcp,:::222->22/tcp, 3000/tcp
gitea-cache     docker-entrypoint.sh redis ...   Up (healthy)   6379/tcp
gitea-traefik   /entrypoint.sh --api --pro ...   Up             0.0.0.0:443->443/tcp,:::443->443/tcp, 0.0.0.0:80->80/tcp,:::80->80/tcp
```

## Installation and Configuration

Head over to the `ROOT_URL` of your gitea installation, in my case it looked like the following:

![gitea-container](https://user-images.githubusercontent.com/567298/138003899-63d0cfd8-ca97-4e9f-b9d6-224987105d03.png)

If you are not automatically redirected to register an account, select "Register" in the top right side:

![gitea-register](https://user-images.githubusercontent.com/567298/138001952-5bf32a22-5287-4d4f-b852-6860dd6417af.png)

If you would like to make use of email, configure your email settings here:

![gitea-email](https://user-images.githubusercontent.com/567298/138002011-f261870d-2fcc-4809-8d12-8317fa252de3.png)

Then configure the admin account:

![gitea-admin-account](https://user-images.githubusercontent.com/567298/138422959-8637e282-7393-4cc2-b942-fe65cf6ea80f.png)

Once you are logged in, you should see the following screen:

![image](https://user-images.githubusercontent.com/567298/138002202-818a364c-74a9-4be9-8963-cd151d10d96f.png)

## SSH Key

Now we would like to create a SSH key so that we can authorize our git client to pull and push to/from Gitea:

```bash
ssh-keygen -f ~/.ssh/gitea-demo -t rsa -C "Gitea-Demo" -q -N ""
```

Then head to your profile, select settings:

![image](https://user-images.githubusercontent.com/567298/138002368-679ca1eb-81a3-4a83-8aaf-e5c182305be4.png)

Select the SSH Tab and select "Add Key":

![image](https://user-images.githubusercontent.com/567298/138002437-a43fbd5b-92a6-40b8-abb1-ebe78c8a5c67.png)

Head back to your terminal and copy your public ssh key from the key that we created earlier:

```bash
cat ~/.ssh/gitea-demo.pub
ssh-rsa AAAAB[x----redacted----x]/en5QDz3vI18n1u4lrKu1YsTR57YL Gitea-Demo
```

Then paste the public key into the form and add the key, you should then see the key present in gitea:

![image](https://user-images.githubusercontent.com/567298/138002601-be049746-fcb6-46ad-a931-bfa11d1725b2.png)

## Create a Git Repository

Now head back to the "Dashboard", then select the "+" sign at the top and create a "New Repository":

![image](https://user-images.githubusercontent.com/567298/138002657-054e9e6a-25bd-4f4a-9bdb-ef368a720e5f.png)

From the repo form, I will be naming my repository "hello-world":

![image](https://user-images.githubusercontent.com/567298/138002751-88ff1793-b6e4-4ac4-886d-d4826e152ffa.png)

Then I selected "Initialise repository with Readme" and I selected to create the repository:

![image](https://user-images.githubusercontent.com/567298/138002819-69ce233d-c073-4ed4-a714-1762d34f5b47.png)

Now when we select the repo, we should see it in the Gitea UI:

![image](https://user-images.githubusercontent.com/567298/138002865-2e9cc6d9-3fea-4f32-91b8-6199063b7d4f.png)

To clone the repository via SSH, select the "SSH" button and click copy to clipboard:

![image](https://user-images.githubusercontent.com/567298/138002962-962b746f-619e-4f08-b058-6d8f4d719e89.png)

Before we clone the repo on our terminal, let's setup the ssh-agent to be active for 1 hour:

```
eval $(ssh-agent -t 3600)
```

Then add the ssh key to the ssh-agent:

```
ssh-add ~/.ssh/gitea-demo

Identity added: ~/.ssh/gitea-demo (Gitea-Demo)
```

*Optional:* If you have a non default ssh key, like the above and you don't want to make use of `ssh-agent` you can setup a SSH Config, for example in `~/.ssh/config`:

```
# Globals
Host *
  StrictHostKeyChecking no
  UserKnownHostsFile /dev/null
  #AddKeysToAgent yes
  #IdentityFile ~/.ssh/id_rsa
  ServerAliveInterval 60
  ServerAliveCountMax 30

# Gitea
Host git.rbkr.xyz
  IdentityFile ~/.ssh/gitea-demo
  User git
  Port 222
```

Now let's clone the repository:

```
git clone ssh://git@git.rbkr.xyz:222/ruanbekker/hello-world.git

Cloning into 'hello-world'...
Warning: Permanently added '[git.rbkr.xyz]:222,[95.x.x.x]:222' (ECDSA) to the list of known hosts.
remote: Enumerating objects: 3, done.
remote: Counting objects: 100% (3/3), done.
remote: Total 3 (delta 0), reused 0 (delta 0), pack-reused 0
Receiving objects: 100% (3/3), done.
```

Change into the directory which we cloned:

```
cd hello-world
```

Let's update the `README.md` file with any content, then after we saved the file, we can see that the file has been changed:

```
git status

On branch master
Your branch is up to date with 'origin/master'.

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
    modified:   README.md

no changes added to commit (use "git add" and/or "git commit -a")
```

Add the file, commit and push to master:

```
git add README.md
git commit -m "Update readme for blogpost"
git push origin master

Writing objects: 100% (3/3), 305 bytes | 305.00 KiB/s, done.
Total 3 (delta 0), reused 0 (delta 0), pack-reused 0
remote: . Processing 1 references
remote: Processed 1 references in total
To ssh://git.rbkr.xyz:222/ruanbekker/hello-world.git
   7804b67..85550dd  master -> master
```

## View the changes

When we head back to the Gitea UI, we can see the README file has been updated, and we can see a git commit sha as well:

![gitea-ui](https://user-images.githubusercontent.com/567298/138003524-0eb7de18-f90d-4671-8d29-1e6614529743.png)

In order to see what changed, we can click on the git commit sha:

![gitea-commit](https://user-images.githubusercontent.com/567298/138003595-262b2f43-06dc-4881-8277-bab17b2ef005.png)

## Swagger API

Gitea ships with Swagger by default and the endpoint is `/api/swagger` which in my case is accessible via:

- https://git.rbkr.xyz/api/swagger

And it looks like the following:

![gitea-swagger](https://user-images.githubusercontent.com/567298/138003950-c9e06f77-3ce6-434e-bf10-a10dcdf68f56.png)

## Thank You

I hope this was helpful, I was really impressed with Gitea. If you liked this content, please make sure to share or come say hi on my website or twitter:

  * **Website**: [ruan.dev](https://ruan.dev)
  * **Twitter**: [@ruanbekker](https://twitter.com/ruanbekker)

Setup Glitchtip Error Monitoring on Docker

Ruan Bekker — Mon, 04 Oct 2021 15:02:18 +0000

Glitchtip is a open-source exception monitoring system, it's similar to Sentry, which collects errors reported from your applications and helps you discover errors in real time and also helps you to understand the health of your applications.

What are we doing today?

In this tutorial we will setup Glitchtip on Docker using Traefik as our Load Balancer and SSL terminations for LetsEncrypt certificates, then we will create a Python Flask application and initiate a error so that we can see how these errors are collected by Glitchtip and we will also add an Webhook Server to demonstrate logging errors to a Webhook Endpoint for further development on errors.

So that we can see our exceptions in our applications like this:

Assumptions

Code Source

I will publish the code for this tutorial under my Github Repository:

https://github.com/ruanbekker/docker-glitchtip-traefik

Environment Details

I have 3 DNS Entries:

Glitchtip: gc.civo.rbkr.xyz
Flask App: flask-app.civo.rbkr.xyz
Webhook: flask-webhook.civo.rbkr.xyz

Which all points to the host running Glitchtip.

Directory Structure

First create the directory structure for our glitchtip demo and change to the directory:

mkdir glitchtip-tutorial
cd glitchtip-tutorial

Create the traefik directory for acme.json where certificate data will be stored, create the file and change permissions on the file:

mkdir traefik
touch traefik/acme.json
chmod 600 traefik/acme.json

Traefik

Open the docker-compose.yml and add the first bit which will Traefik, ensure that you replace me@example.com with your email under certificatesResolvers.letsencrypt.acme.email:

version: '3.8'

x-logging:
  &default-logging
  driver: "json-file"
  options:
    max-size: "1m"

services:
  glitchtip-traefik:
    image: traefik:2.4
    container_name: glitchtip-traefik
    restart: unless-stopped
    volumes:
      - ./traefik/acme.json:/acme.json
      - /var/run/docker.sock:/var/run/docker.sock
    networks:
      - glitchtip
    ports:
      - 80:80
      - 443:443
    command:
      - '--api'
      - '--providers.docker=true'
      - '--providers.docker.exposedByDefault=false'
      - '--entrypoints.http=true'
      - '--entrypoints.http.address=:80'
      - '--entrypoints.http.http.redirections.entrypoint.to=https'
      - '--entrypoints.http.http.redirections.entrypoint.scheme=https'
      - '--entrypoints.https=true'
      - '--entrypoints.https.address=:443'
      - '--certificatesResolvers.letsencrypt.acme.email=me@example.com'
      - '--certificatesResolvers.letsencrypt.acme.storage=acme.json'
      - '--certificatesResolvers.letsencrypt.acme.httpChallenge.entryPoint=http'
      - '--log=true'
      - '--log.level=INFO'
    logging:
      driver: "json-file"
      options:
        max-size: "1m"

networks:
  glitchtip:
    name: glitchtip

You can now start Traefik so long:

docker-compose up -d

Glitchtip

Now we will add the Glitchtip components, you can refer to their documentation if you want to read more into these configuration options. Add the following services below glitchtip-traefk:

  glitchtip-postgres:
    image: postgres:13
    container_name: glitchtip-postgres
    restart: unless-stopped
    environment:
      POSTGRES_HOST_AUTH_METHOD: "trust"
      POSTGRES_DB: postgres
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: postgres
      PGDATA: /var/lib/postgresql/data/pgdata
    volumes:
      - ./data/postgres:/var/lib/postgresql/data
    networks:
      - glitchtip
    healthcheck:
      test: ["CMD", "pg_isready", "-U", "postgres"]
      interval: 5s
      retries: 5
    logging: *default-logging

  glitchtip-redis:
    image: redis
    container_name: glitchtip-redis
    restart: unless-stopped
    networks:
      - glitchtip
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 5s
      timeout: 30s
      retries: 50
    logging: *default-logging

  glitchtip-web:
    image: glitchtip/glitchtip
    container_name: glitchtip-web
    restart: unless-stopped
    depends_on:
      glitchtip-traefik:
        condition: service_started
      glitchtip-postgres:
        condition: service_healthy
      glitchtip-redis:
        condition: service_healthy
    environment:
      DATABASE_URL: postgres://postgres:postgres@glitchtip-postgres:5432/postgres
      REDIS_URL: redis://glitchtip-redis:6379/0
      SECRET_KEY: jsdf93892309fhufhr
      PORT: 8000
      EMAIL_URL: "${EMAIL_URL}"
      DEFAULT_FROM_EMAIL: ${EMAIL_FROM_ADDRESS}
      GLITCHTIP_DOMAIN: "https://gt.civo.rbkr.xyz"
      ENABLE_OPEN_USER_REGISTRATION: "True"
      GLITCHTIP_MAX_EVENT_LIFE_DAYS: 90
    networks:
      - glitchtip
    labels:
      - 'traefik.enable=true'
      - 'traefik.http.routers.glitchtip.rule=Host(`gt.civo.rbkr.xyz`)'
      - 'traefik.http.routers.glitchtip.entrypoints=https'
      - 'traefik.http.routers.glitchtip.tls=true'
      - 'traefik.http.routers.glitchtip.tls.certresolver=letsencrypt'
      - 'traefik.http.routers.glitchtip.service=glitchtip-service'
      - 'traefik.http.services.glitchtip-service.loadbalancer.server.port=8000'
    logging: *default-logging

  glitchtip-worker:
    image: glitchtip/glitchtip
    container_name: glitchtip-worker
    restart: unless-stopped
    command: celery -A glitchtip worker -B -l INFO
    depends_on:
      glitchtip-postgres:
        condition: service_healthy
      glitchtip-redis:
        condition: service_healthy
    environment:
      DATABASE_URL: postgres://postgres:postgres@glitchtip-postgres:5432/postgres
      REDIS_URL: redis://glitchtip-redis:6379/0
      SECRET_KEY: jsdf93892309fhufhr
      PORT: 8000
      EMAIL_URL: "${EMAIL_URL}"
      DEFAULT_FROM_EMAIL: ${EMAIL_FROM_ADDRESS}
      GLITCHTIP_DOMAIN: "https://gt.civo.rbkr.xyz"
    networks:
      - glitchtip
    logging: *default-logging

  glitchtip-migrate:
    image: glitchtip/glitchtip
    container_name: glitchtip-migrate
    depends_on:
      glitchtip-postgres:
        condition: service_healthy
      glitchtip-redis:
        condition: service_healthy
    command: "./manage.py migrate"
    environment:
      DATABASE_URL: postgres://postgres:postgres@glitchtip-postgres:5432/postgres
      REDIS_URL: redis://glitchtip-redis:6379/0
      SECRET_KEY: jsdf93892309fhufhr
      PORT: 8000
      EMAIL_URL: "${EMAIL_URL}"
      DEFAULT_FROM_EMAIL: ${EMAIL_FROM_ADDRESS}
      GLITCHTIP_DOMAIN: "https://gt.civo.rbkr.xyz"
    networks:
      - glitchtip
    logging: *default-logging

Ensure that you change the following:

EMAIL_URL eg. smtp://apikey:your-password@smtp.sendgrid.net:587 (sendgrid offers free mail delivery)
DEFAULT_FROM_EMAIL eg. no-reply@yourdomain.com
Your FQDN for glitchtip in 'traefik.http.routers.glitchtip.rule=Host()'

Once your configuration is updated, start glitchtip:

docker-compose pull
docker-compose up -d

Once all the services are started you should be able to access the UI and will look more or less like this:

Since ENABLE_OPEN_USER_REGISTRATION is set to True, we can register our user:

Once you register you will be logged in and on the organization page to create a organization:

For this tutorial, I will be using the name my-org:

I will go ahead and create a application in my organisation called flask-service-dev and I will be selecting Python Flask for this tutorial, and I want to associate this project to a team called engineering, at the bottom select team and create your team:

Once the team has been created and selected, select the framework and name your service:

Once you created your project you will get a getting started guide on how to configure your application:

On the right hand side you will get the configuration values, which we will use as environment variables on our application that we want to emit errors to glitchtip:

https://6ee24c1a446347b99e9574edfe990393@gt.civo.rbkr.xyz/1
https://gt.civo.rbkr.xyz/api/1/security/?glitchtip_key=6ee24c1a446347b99e9574edfe990393

Now that we have signed up we can disable ENABLE_OPEN_USER_REGISTRATION by setting it to False in the docker-compose.yml, then run docker-compose up -d.

Optional: Glitchtip is written in Django, so you can also use the following method to manage users and access the admin panel on /admin:

$ docker-compose run glitchtip-migrate ./manage.py createsuperuser
Creating glitchdip_glitchtip-migrate_run ... done
Email: me@example.com
Password:
Password (again):
Superuser created successfully.

Python Flask App

Now we will create the application that we want to monitor for errors on glitchtip. From the glitchtip-tutorial directory, create the application code and dockerfile for our python flask app:

mkdir flask-app
touch flask-app/requirements.txt
touch flask-app/app.py
touch flask-app/Dockerfile

First add the requirements in flask-app/requirements.txt:

flask==1.1.2
sentry-sdk[flask]
gunicorn==20.0.4

Next our dockerfile in flask-app/Dockerfile:

FROM python:3.8-slim
ADD requirements.txt /src/requirements.txt
ADD run.sh /src/run.sh
RUN chmod +x /src/run.sh
RUN pip install --upgrade -r /src/requirements.txt
ADD app.py /src/app.py
CMD ["/src/run.sh"]

Our boot script that will start flask using gunicorn in flask-app/run.sh:

#!/bin/sh
gunicorn app:app \
    --workers 2 \
    --threads 2 \
    --bind 0.0.0.0:80 \
    --capture-output \
    --access-logfile '-' \
    --error-logfile '-'

Then our application code in flask-app/app.py:

import os
import logging
from flask import Flask
import sentry_sdk
from sentry_sdk.integrations.flask import FlaskIntegration

GLITCHTIP_DSN = os.environ['DSN']

sentry_sdk.init(
    dsn=GLITCHTIP_DSN,
    integrations=[FlaskIntegration()]
)

app = Flask(__name__)
app.config['DEBUG'] = False

@app.before_first_request
def setup_logging():
    app.logger.addHandler(logging.StreamHandler())
    app.logger.setLevel(logging.INFO)
    app.logger.info('logs enabled')

@app.route('/')
def root():
    return 'hello, world'

@app.route('/log-error')
def log_error():
    app.logger.error('logging error with glitchtip')
    return 'logged error'

@app.route('/trigger-error')
def trigger_error():
    division_by_zero = 1 / 0

if __name__ == '__main__':
    app.logger.info('starting server')
    app.run()

Then we add this section to our docker-compose.yml:

  glitchtip-flask-app:
    build:
      context: flask-app
      dockerfile: Dockerfile
    container_name: glitchtip-flask-app
    restart: unless-stopped
    depends_on:
      glitchtip-traefik:
        condition: service_started
    environment:
      DSN: "https://6ee24c1a446347b99e9574edfe990393@gt.civo.rbkr.xyz/1"
    networks:
      - glitchtip
    labels:
      - 'traefik.enable=true'
      - 'traefik.http.routers.app.rule=Host(`flask-app.civo.rbkr.xyz`)'
      - 'traefik.http.routers.app.entrypoints=https'
      - 'traefik.http.routers.app.tls=true'
      - 'traefik.http.routers.app.tls.certresolver=letsencrypt'
      - 'traefik.http.routers.app.service=app-service'
      - 'traefik.http.services.app-service.loadbalancer.server.port=80'
    logging: *default-logging

So now our full docker-compose.yml should look like this more or less:

---
version: '3.8'

x-logging:
  &default-logging
  driver: "json-file"
  options:
    max-size: "1m"

services:
  glitchtip-traefik:
    image: traefik:2.4
    container_name: glitchtip-traefik
    restart: unless-stopped
    volumes:
      - ./traefik/acme.json:/acme.json
      - /var/run/docker.sock:/var/run/docker.sock
    networks:
      - glitchtip
    ports:
      - 80:80
      - 443:443
    command:
      - '--api'
      - '--providers.docker=true'
      - '--providers.docker.exposedByDefault=false'
      - '--entrypoints.http=true'
      - '--entrypoints.http.address=:80'
      - '--entrypoints.http.http.redirections.entrypoint.to=https'
      - '--entrypoints.http.http.redirections.entrypoint.scheme=https'
      - '--entrypoints.https=true'
      - '--entrypoints.https.address=:443'
      - '--certificatesResolvers.letsencrypt.acme.email=me@example.com'
      - '--certificatesResolvers.letsencrypt.acme.storage=acme.json'
      - '--certificatesResolvers.letsencrypt.acme.httpChallenge.entryPoint=http'
      - '--log=true'
      - '--log.level=INFO'
    logging:
      driver: "json-file"
      options:
        max-size: "1m"

  glitchtip-postgres:
    image: postgres:13
    container_name: glitchtip-postgres
    restart: unless-stopped
    environment:
      POSTGRES_HOST_AUTH_METHOD: "trust"
      POSTGRES_DB: postgres
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: postgres
      PGDATA: /var/lib/postgresql/data/pgdata
    volumes:
      - ./data/postgres:/var/lib/postgresql/data
    networks:
      - glitchtip
    healthcheck:
      test: ["CMD", "pg_isready", "-U", "postgres"]
      interval: 5s
      retries: 5
    logging: *default-logging

  glitchtip-redis:
    image: redis
    container_name: glitchtip-redis
    restart: unless-stopped
    networks:
      - glitchtip
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 5s
      timeout: 30s
      retries: 50
    logging: *default-logging

  glitchtip-web:
    image: glitchtip/glitchtip
    container_name: glitchtip-web
    restart: unless-stopped
    depends_on:
      glitchtip-traefik:
        condition: service_started
      glitchtip-postgres:
        condition: service_healthy
      glitchtip-redis:
        condition: service_healthy
    environment:
      DATABASE_URL: postgres://postgres:postgres@glitchtip-postgres:5432/postgres
      REDIS_URL: redis://glitchtip-redis:6379/0
      SECRET_KEY: jsdf93892309fhufhr
      PORT: 8000
      EMAIL_URL: "${EMAIL_URL}"
      DEFAULT_FROM_EMAIL: ${EMAIL_FROM_ADDRESS}
      GLITCHTIP_DOMAIN: "https://gt.civo.rbkr.xyz"
      ENABLE_OPEN_USER_REGISTRATION: "False"
      GLITCHTIP_MAX_EVENT_LIFE_DAYS: 90
    networks:
      - glitchtip
    labels:
      - 'traefik.enable=true'
      - 'traefik.http.routers.glitchtip.rule=Host(`gt.civo.rbkr.xyz`)'
      - 'traefik.http.routers.glitchtip.entrypoints=https'
      - 'traefik.http.routers.glitchtip.tls=true'
      - 'traefik.http.routers.glitchtip.tls.certresolver=letsencrypt'
      - 'traefik.http.routers.glitchtip.service=glitchtip-service'
      - 'traefik.http.services.glitchtip-service.loadbalancer.server.port=8000'
    logging: *default-logging

  glitchtip-worker:
    image: glitchtip/glitchtip
    container_name: glitchtip-worker
    restart: unless-stopped
    command: celery -A glitchtip worker -B -l INFO
    depends_on:
      glitchtip-postgres:
        condition: service_healthy
      glitchtip-redis:
        condition: service_healthy
    environment:
      DATABASE_URL: postgres://postgres:postgres@glitchtip-postgres:5432/postgres
      REDIS_URL: redis://glitchtip-redis:6379/0
      SECRET_KEY: jsdf93892309fhufhr
      PORT: 8000
      EMAIL_URL: "${EMAIL_URL}"
      DEFAULT_FROM_EMAIL: ${EMAIL_FROM_ADDRESS}
    networks:
      - glitchtip
    logging: *default-logging

  glitchtip-migrate:
    image: glitchtip/glitchtip
    container_name: glitchtip-migrate
    depends_on:
      glitchtip-postgres:
        condition: service_healthy
      glitchtip-redis:
        condition: service_healthy
    command: "./manage.py migrate"
    environment:
      DATABASE_URL: postgres://postgres:postgres@glitchtip-postgres:5432/postgres
      REDIS_URL: redis://glitchtip-redis:6379/0
      SECRET_KEY: jsdf93892309fhufhr
      PORT: 8000
      EMAIL_URL: "${EMAIL_URL}"
      DEFAULT_FROM_EMAIL: ${EMAIL_FROM_ADDRESS}
      GLITCHTIP_DOMAIN: "https://gt.civo.rbkr.xyz"
    networks:
      - glitchtip
    logging: *default-logging

  glitchtip-flask-app:
    build:
      context: flask-app
      dockerfile: Dockerfile
    container_name: glitchtip-flask-app
    restart: unless-stopped
    depends_on:
      glitchtip-traefik:
        condition: service_started
    environment:
      DSN: "https://6ee24c1a446347b99e9574edfe990393@gt.civo.rbkr.xyz/1"
    networks:
      - glitchtip
    labels:
      - 'traefik.enable=true'
      - 'traefik.http.routers.app.rule=Host(`flask-app.civo.rbkr.xyz`)'
      - 'traefik.http.routers.app.entrypoints=https'
      - 'traefik.http.routers.app.tls=true'
      - 'traefik.http.routers.app.tls.certresolver=letsencrypt'
      - 'traefik.http.routers.app.service=app-service'
      - 'traefik.http.services.app-service.loadbalancer.server.port=80'
    logging: *default-logging

networks:
  glitchtip:
    name: glitchtip

Build the flask-app container and start the service:

docker-compose up --build -d

Making a test request:

$ curl -IL http://flask-app.civo.rbkr.xyz
HTTP/1.1 308 Permanent Redirect
Location: https://flask-app.civo.rbkr.xyz/
Date: Tue, 28 Sep 2021 06:00:27 GMT
Content-Length: 18
Content-Type: text/plain; charset=utf-8

HTTP/2 200
content-type: text/html; charset=utf-8
date: Tue, 28 Sep 2021 06:00:27 GMT
server: gunicorn/20.0.4
content-length: 12

Logging Exceptions to Glitchtip

Now let's access the endpoints that will cause an exception, let's first make a request to /trigger-error, then head over to "Issues", or if you have multiple projects, select your organization, then select your project's issues:

From the issues view, we can see we had one issue:

When we select the issue, we can see more information about this exception, such as the useragent info, the request path, where in our application the exception was raised etc:

As you can see, Glitchtip also provide information deeper in the stack where the exception was raised from:

There's a lot of information, but what I also found nice, was that it logs span_id and trace_id, the sdk version etc:

If we have a look at the /log-error, which will just log an error event, from the issues view, we can see we had one issue:

When we select the issue, we can see more information about this exception, such as the useragent info, the request path, etc:

Alerting

We can setup alerting for every time we receive an exception, to do so, head over to "Settings", "Projects" and you will find "Project Alerts":

When we select "Create New Alert", you will get the alert configuration and by default the alert will be configured to the email address of the team members of the project.

If we "Add Alert Recipient", you will see that we have the option to add a "Webhook URL", so let's go back to our docker-compose.yml and setup a Python Flask Webhook:

  glitchtip-flask-webhook:
    build:
      context: flask-webhook
      dockerfile: Dockerfile
    container_name: glitchtip-flask-webhook
    restart: unless-stopped
    depends_on:
      glitchtip-traefik:
        condition: service_started
    networks:
      - glitchtip
    labels:
      - 'traefik.enable=true'
      - 'traefik.http.routers.webhook.rule=Host(`flask-webhook.civo.rbkr.xyz`)'
      - 'traefik.http.routers.webhook.entrypoints=https'
      - 'traefik.http.routers.webhook.tls=true'
      - 'traefik.http.routers.webhook.tls.certresolver=letsencrypt'
      - 'traefik.http.routers.webhook.service=webhook-service'
      - 'traefik.http.services.webhook-service.loadbalancer.server.port=80'
    logging: *default-logging
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:80/health"]
      interval: 5s
      timeout: 3s
      retries: 60

Then we create our flask-webhook directory:

mkdir flask-webhook

Create the required files, first our application flask-webhook/app.py:

import logging
from flask import Flask, request, jsonify

app = Flask(__name__)
app.config['DEBUG'] = False

@app.before_first_request
def setup_logging():
    app.logger.addHandler(logging.StreamHandler())
    app.logger.setLevel(logging.INFO)

@app.route('/', methods=['POST'])
def webhook():
    payload = request.get_json(force=True)
    app.logger.info(payload)
    return jsonify(payload)

@app.route('/health', methods=['GET'])
def health():
    payload = {'status': 'up'}
    return jsonify(payload)

if __name__ == '__main__':
    app.logger.info('starting server')
    app.run()

Then our flask-webhook/Dockerfile:

FROM python:3.8-slim
RUN apt update && apt install curl -y
WORKDIR /src
ADD requirements.txt /src/requirements.txt
ADD run.sh /src/run.sh
RUN chmod +x /src/run.sh
RUN pip install --upgrade -r /src/requirements.txt
ADD app.py /src/app.py

CMD ["/src/run.sh"]

Then our flask-webhook/requirements.txt:

flask==1.1.2
gunicorn==20.0.4

Then when everything is in place, build and start the container:

docker-compose up --build -d

Once the application has started, from Settings, Projects, Project Alerts, when we add a new alert recipient, we select the webhook as a recipient type and add our webhook url:

Now we should see:

Select submit and then invoke the /trigger-error url:

$ curl -I https://flask-app.civo.rbkr.xyz/trigger-error
HTTP/2 500
content-type: text/html; charset=utf-8
date: Tue, 28 Sep 2021 09:36:29 GMT
server: gunicorn/20.0.4
content-length: 290

Now when we look at the logs of our webhook container:

$ docker-compose logs -f glitchtip-flask-webhook
...
glitchtip-flask-webhook    | {'alias': 'GlitchTip', 'text': 'GlitchTip Alert', 'attachments': [{'title': 'ZeroDivisionError: division by zero', 'title_link': 'https://gt.civo.rbkr.xyz/my-org/issues/2', 'text': 'trigger_error', 'image_url': None, 'color': '#e52b50'}], 'sections': [{'activityTitle': 'ZeroDivisionError: division by zero', 'activitySubtitle': '[View Issue FLASK-SERVICE-DEV-1](https://gt.civo.rbkr.xyz/my-org/issues/2)'}]}

If we prettify it, you can see we can consume this information and do our own integrations:

>>> print(json.dumps(e, indent=2))
{
  "alias": "GlitchTip",
  "text": "GlitchTip Alert",
  "attachments": [
    {
      "title": "ZeroDivisionError: division by zero",
      "title_link": "https://gt.civo.rbkr.xyz/my-org/issues/2",
      "text": "trigger_error",
      "image_url": null,
      "color": "#e52b50"
    }
  ],
  "sections": [
    {
      "activityTitle": "ZeroDivisionError: division by zero",
      "activitySubtitle": "[View Issue FLASK-SERVICE-DEV-1](https://gt.civo.rbkr.xyz/my-org/issues/2)"
    }
  ]
}

And our email notification will look like this:

The Complete Compose

This will be published to my github repository, but the complete docker-compose.yml:

---
version: '3.8'

x-logging:
  &default-logging
  driver: "json-file"
  options:
    max-size: "1m"

services:
  glitchtip-traefik:
    image: traefik:2.4
    container_name: glitchtip-traefik
    restart: unless-stopped
    volumes:
      - ./traefik/acme.json:/acme.json
      - /var/run/docker.sock:/var/run/docker.sock
    networks:
      - glitchtip
    ports:
      - 80:80
      - 443:443
    command:
      - '--api'
      - '--providers.docker=true'
      - '--providers.docker.exposedByDefault=false'
      - '--entrypoints.http=true'
      - '--entrypoints.http.address=:80'
      - '--entrypoints.http.http.redirections.entrypoint.to=https'
      - '--entrypoints.http.http.redirections.entrypoint.scheme=https'
      - '--entrypoints.https=true'
      - '--entrypoints.https.address=:443'
      - '--certificatesResolvers.letsencrypt.acme.email=email@example.com'
      - '--certificatesResolvers.letsencrypt.acme.storage=acme.json'
      - '--certificatesResolvers.letsencrypt.acme.httpChallenge.entryPoint=http'
      - '--log=true'
      - '--log.level=INFO'
    logging:
      driver: "json-file"
      options:
        max-size: "1m"

  glitchtip-postgres:
    image: postgres:13
    container_name: glitchtip-postgres
    restart: unless-stopped
    environment:
      POSTGRES_HOST_AUTH_METHOD: "trust"
      POSTGRES_DB: postgres
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: postgres
      PGDATA: /var/lib/postgresql/data/pgdata
    volumes:
      - ./data/postgres:/var/lib/postgresql/data
    networks:
      - glitchtip
    healthcheck:
      test: ["CMD", "pg_isready", "-U", "postgres"]
      interval: 5s
      retries: 5
    logging: *default-logging

  glitchtip-redis:
    image: redis
    container_name: glitchtip-redis
    restart: unless-stopped
    networks:
      - glitchtip
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 5s
      timeout: 30s
      retries: 50
    logging: *default-logging

  glitchtip-web:
    image: glitchtip/glitchtip
    container_name: glitchtip-web
    restart: unless-stopped
    depends_on:
      glitchtip-traefik:
        condition: service_started
      glitchtip-postgres:
        condition: service_healthy
      glitchtip-redis:
        condition: service_healthy
    environment:
      DATABASE_URL: postgres://postgres:postgres@glitchtip-postgres:5432/postgres
      REDIS_URL: redis://glitchtip-redis:6379/0
      SECRET_KEY: jsdf93892309fhufhr
      PORT: 8000
      EMAIL_URL: "${EMAIL_URL}"
      DEFAULT_FROM_EMAIL: ${EMAIL_FROM_ADDRESS}
      GLITCHTIP_DOMAIN: "https://gt.civo.rbkr.xyz"
      ENABLE_OPEN_USER_REGISTRATION: "False"
      GLITCHTIP_MAX_EVENT_LIFE_DAYS: 90
    networks:
      - glitchtip
    labels:
      - 'traefik.enable=true'
      - 'traefik.http.routers.glitchtip.rule=Host(`gt.civo.rbkr.xyz`)'
      - 'traefik.http.routers.glitchtip.entrypoints=https'
      - 'traefik.http.routers.glitchtip.tls=true'
      - 'traefik.http.routers.glitchtip.tls.certresolver=letsencrypt'
      - 'traefik.http.routers.glitchtip.service=glitchtip-service'
      - 'traefik.http.services.glitchtip-service.loadbalancer.server.port=8000'
    logging: *default-logging

  glitchtip-worker:
    image: glitchtip/glitchtip
    container_name: glitchtip-worker
    restart: unless-stopped
    command: celery -A glitchtip worker -B -l INFO
    depends_on:
      glitchtip-postgres:
        condition: service_healthy
      glitchtip-redis:
        condition: service_healthy
    environment:
      DATABASE_URL: postgres://postgres:postgres@glitchtip-postgres:5432/postgres
      REDIS_URL: redis://glitchtip-redis:6379/0
      SECRET_KEY: jsdf93892309fhufhr
      PORT: 8000
      EMAIL_URL: "${EMAIL_URL}"
      DEFAULT_FROM_EMAIL: ${EMAIL_FROM_ADDRESS}
      GLITCHTIP_DOMAIN: "https://gt.civo.rbkr.xyz"
    networks:
      - glitchtip
    logging: *default-logging

  glitchtip-migrate:
    image: glitchtip/glitchtip
    container_name: glitchtip-migrate
    depends_on:
      glitchtip-postgres:
        condition: service_healthy
      glitchtip-redis:
        condition: service_healthy
    command: "./manage.py migrate"
    environment:
      DATABASE_URL: postgres://postgres:postgres@glitchtip-postgres:5432/postgres
      REDIS_URL: redis://glitchtip-redis:6379/0
      SECRET_KEY: jsdf93892309fhufhr
      PORT: 8000
      EMAIL_URL: "${EMAIL_URL}"
      DEFAULT_FROM_EMAIL: ${EMAIL_FROM_ADDRESS}
      GLITCHTIP_DOMAIN: "https://gt.civo.rbkr.xyz"
    networks:
      - glitchtip
    logging: *default-logging

  glitchtip-flask-app:
    build:
      context: flask-app
      dockerfile: Dockerfile
    container_name: glitchtip-flask-app
    restart: unless-stopped
    depends_on:
      glitchtip-traefik:
        condition: service_started
    environment:
      DSN: "https://6ee24c1a446347b99e9574edfe990393@gt.civo.rbkr.xyz/1"
    networks:
      - glitchtip
    labels:
      - 'traefik.enable=true'
      - 'traefik.http.routers.app.rule=Host(`flask-app.civo.rbkr.xyz`)'
      - 'traefik.http.routers.app.entrypoints=https'
      - 'traefik.http.routers.app.tls=true'
      - 'traefik.http.routers.app.tls.certresolver=letsencrypt'
      - 'traefik.http.routers.app.service=app-service'
      - 'traefik.http.services.app-service.loadbalancer.server.port=80'
    logging: *default-logging

  glitchtip-flask-webhook:
    build:
      context: flask-webhook
      dockerfile: Dockerfile
    container_name: glitchtip-flask-webhook
    restart: unless-stopped
    depends_on:
      glitchtip-traefik:
        condition: service_started
    networks:
      - glitchtip
    labels:
      - 'traefik.enable=true'
      - 'traefik.http.routers.webhook.rule=Host(`flask-webhook.civo.rbkr.xyz`)'
      - 'traefik.http.routers.webhook.entrypoints=https'
      - 'traefik.http.routers.webhook.tls=true'
      - 'traefik.http.routers.webhook.tls.certresolver=letsencrypt'
      - 'traefik.http.routers.webhook.service=webhook-service'
      - 'traefik.http.services.webhook-service.loadbalancer.server.port=80'
    logging: *default-logging
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:80/health"]
      interval: 5s
      timeout: 3s
      retries: 60

networks:
  glitchtip:
    name: glitchtip

I left the domain endpoints and glitchtip application keys in the code intentionally for demonstration purposes, but deleted the stack after the time of writing.

Thank You

I hope this was helpful, I was really impressed with Glitchtip. If you liked this content, please make sure to share or come say hi on my website or twitter:

Website: ruan.dev
Twitter: @ruanbekker

Interacting with your Bitcoin Node

Ruan Bekker — Mon, 13 Sep 2021 07:24:36 +0000

This is a series of posts on running your own bitcoin full-node on the testnet chain.

In my previous post I demonstrated how to setup a bitcoin full-node on the testnet chain.

In this post we will:

interact with our node using the CLI and JSON-RPC
create wallets and accounts
send test bitcoin to our accounts
track transactions using a block explorer
and more.

Pre-Requisites

Bitcoin core should be installed and fully synced, you can reference my previous post if you want to follow this guide.

CLI Usage

To get the current block count:

$ bitcoin-cli getblockcount
2091215

To get some basic info about the first block ever created on the bitcoin blockchain. As the genesis block, it has the index value 0. We can use getblockhas to get the hash value for the first block ever created:

$ bitcoin-cli getblockhash 0
000000000933ea01ad0ee984209779baaec3ced90fa3f408719526f8d77f4943

We can now use getblock with the hash value to retrieve details about the block:

$ bitcoin-cli getblock 000000000933ea01ad0ee984209779baaec3ced90fa3f408719526f8d77f4943
{
  "hash": "000000000933ea01ad0ee984209779baaec3ced90fa3f408719526f8d77f4943",
  "confirmations": 2004218,
  "strippedsize": 285,
  "size": 285,
  "weight": 1140,
  "height": 0,
  "version": 1,
  "versionHex": "00000001",
  "merkleroot": "4a5e1e4baab89f3a32518a88c31bc87f618f76673e2cc77ab2127b7afdeda33b",
  "tx": [
    "4a5e1e4baab89f3a32518a88c31bc87f618f76673e2cc77ab2127b7afdeda33b"
  ],
  "time": 1296688602,
  "mediantime": 1296688602,
  "nonce": 414098458,
  "bits": "1d00ffff",
  "difficulty": 1,
  "chainwork": "0000000000000000000000000000000000000000000000000000000100010001",
  "nTx": 1,
  "nextblockhash": "00000000b873e79784647a6c82962c70d228557d24a747ea4d1b8bbe878e1206"
}

We can use the nextblockhash to retrieve information about block 1:

$ bitcoin-cli getblock 00000000b873e79784647a6c82962c70d228557d24a747ea4d1b8bbe878e1206
{
  "hash": "00000000b873e79784647a6c82962c70d228557d24a747ea4d1b8bbe878e1206",
  "confirmations": 2004217,
  "strippedsize": 190,
  "size": 190,
  "weight": 760,
  "height": 1,
  "version": 1,
  "versionHex": "00000001",
  "merkleroot": "f0315ffc38709d70ad5647e22048358dd3745f3ce3874223c80a7c92fab0c8ba",
  "tx": [
    "f0315ffc38709d70ad5647e22048358dd3745f3ce3874223c80a7c92fab0c8ba"
  ],
  "time": 1296688928,
  "mediantime": 1296688928,
  "nonce": 1924588547,
  "bits": "1d00ffff",
  "difficulty": 1,
  "chainwork": "0000000000000000000000000000000000000000000000000000000200020002",
  "nTx": 1,
  "previousblockhash": "000000000933ea01ad0ee984209779baaec3ced90fa3f408719526f8d77f4943",
  "nextblockhash": "000000006c02c8ea6e4ff69651f7fcde348fb9d557a06e6957b65552002a7820"
}

We can use -getinfo to get information show as the verification progress and balances in our wallets, should they exist:

$ bitcoin-cli -getinfo
{
  "version": 210100,
  "blocks": 2091215,
  "headers": 2091215,
  "verificationprogress": 0.9999993040419591,
  "timeoffset": 0,
  "connections": {
    "in": 0,
    "out": 10,
    "total": 10
  },
  "proxy": "",
  "difficulty": 16777216,
  "chain": "test",
  "relayfee": 0.00001000,
  "warnings": "Warning: unknown new rules activated (versionbit 28)",
  "balances": {
  }
}

For more examples view chainquery

JSON RPC Usage

First get the jsonrpc user and password:

$ cat ~/.bitcoin/bitcoin.conf | grep -E '(rpcuser|rpcpassword)'
rpcuser=bitcoin
rpcpassword=xxxxxxxxxxxxxx

Because this is a test environment, we can set the username and password as an environment variable:

# the user and password might differ on your setup
$ export bitcoinauth="bitcoin:xxxxxxxxxxxxxx"

Wallet Interaction

Create a wallet:

curl -u "$bitcoinauth" -d '{"jsonrpc": "1.0", "id": "curltest", "method": "createwallet", "params": ["wallet"]}' -H 'content-type: text/plain;' http://127.0.0.1:18332/
{"result":{"name":"wallet","warning":""},"error":null,"id":"curltest"}

List wallets:

curl -u "$bitcoinauth" -d '{"jsonrpc": "1.0", "id": "tutorial", "method": "listwallets", "params": []}' -H 'content-type: text/plain;' http://127.0.0.1:18332/
{"result":["wallet"],"error":null,"id":"tutorial"}

If we inspect the bitcoin.conf we will notice that we don't have the wallet loaded in our config:

$ cat ~/.bitcoin/bitcoin.conf | grep -c 'wallet='
0

But since we created the wallet, we can see the wallet.dat in our data directory:

$ find /blockchain/ -type f -name wallet.dat
/blockchain/bitcoin/data/testnet3/wallets/wallet/wallet.dat

Let's restart the bitcoind service and see if we can still list our wallet, first restart the service:

$ sudo systemctl restart bitcoind

Then wait a couple of seconds and list the wallets:

$ curl -u "$bitcoinauth" -d '{"jsonrpc": "1.0", "id": "tutorial", "method": "listwallets", "params": []}' -H 'content-type: text/plain;' http://127.0.0.1:18332/
{"result":[],"error":null,"id":"tutorial"}

As you can see it's not loaded, due to it not being in the config. Let's update our config ~/.bitcoin/bitcoin.conf, and restart the service:

# more wallets can be referenced by using another wallet= config
[test]
wallet=wallet
# which corresponds to datadir + walletdir
# /blockchain/bitcoin/data/testnet3/wallets/wallet/wallet.dat
# /blockchain/bitcoin/data/testnet3/wallets/wallet/db.log

When I restarted the bitcoind service, I checked the logs with journalctl -fu bitcoind, and I could see the wallet has been loaded:

Jun 10 13:16:23 ip-172-31-82-15 bitcoind[41053]: 2021-06-10T13:16:23Z init message: Loading wallet...
Jun 10 13:16:23 ip-172-31-82-15 bitcoind[41053]: 2021-06-10T13:16:23Z BerkeleyEnvironment::Open: LogDir=/blockchain/bitcoin/data/testnet3/wallets/wallet/database ErrorFile=/blockchain/bitcoin/data/testnet3/wallets/wallet//db.log
Jun 10 13:16:23 ip-172-31-82-15 bitcoind[41053]: 2021-06-10T13:16:23Z txindex thread start
Jun 10 13:16:23 ip-172-31-82-15 bitcoind[41053]: 2021-06-10T13:16:23Z txindex is enabled at height 2004267
Jun 10 13:16:23 ip-172-31-82-15 bitcoind[41053]: 2021-06-10T13:16:23Z txindex thread exit
Jun 10 13:16:23 ip-172-31-82-15 bitcoind[41053]: 2021-06-10T13:16:23Z [wallet] Wallet File Version = 169900
Jun 10 13:16:23 ip-172-31-82-15 bitcoind[41053]: 2021-06-10T13:16:23Z [wallet] Keys: 2001 plaintext, 0 encrypted, 2001 w/ metadata, 2001 total. Unknown wallet records: 0
Jun 10 13:16:23 ip-172-31-82-15 bitcoind[41053]: 2021-06-10T13:16:23Z [wallet] Wallet completed loading in              39ms
Jun 10 13:16:23 ip-172-31-82-15 bitcoind[41053]: 2021-06-10T13:16:23Z init message: Rescanning...
Jun 10 13:16:23 ip-172-31-82-15 bitcoind[41053]: 2021-06-10T13:16:23Z [wallet] Rescanning last 2 blocks (from block 2004265)...
Jun 10 13:16:23 ip-172-31-82-15 bitcoind[41053]: 2021-06-10T13:16:23Z [wallet] Rescan started from block 000000000000001f778a8e2b68cf05490ae000e653b925bb0552c1b79ef4fe70...
Jun 10 13:16:23 ip-172-31-82-15 bitcoind[41053]: 2021-06-10T13:16:23Z [wallet] Rescan completed in               2ms
Jun 10 13:16:23 ip-172-31-82-15 bitcoind[41053]: 2021-06-10T13:16:23Z [wallet] setKeyPool.size() = 2000
Jun 10 13:16:23 ip-172-31-82-15 bitcoind[41053]: 2021-06-10T13:16:23Z [wallet] mapWallet.size() = 0
Jun 10 13:16:23 ip-172-31-82-15 bitcoind[41053]: 2021-06-10T13:16:23Z [wallet] m_address_book.size() = 0

When I list the wallets again:

$ curl -u "$bitcoinauth" -d '{"jsonrpc": "1.0", "id": "tutorial", "method": "listwallets", "params": []}' -H 'content-type: text/plain;' http://127.0.0.1:18332/
{"result":["wallet"],"error":null,"id":"tutorial"}

Now that the wallet has been loaded, we can get wallet info:

$ curl -s -u "$bitcoinauth" -d '{"jsonrpc": "1.0", "id": "tutorial", "method": "getwalletinfo", "params": []}' -H 'content-type: text/plain;' http://127.0.0.1:18332/ | jq .
{
  "result": {
    "walletname": "wallet",
    "walletversion": 169900,
    "format": "bdb",
    "balance": 0,
    "unconfirmed_balance": 0,
    "immature_balance": 0,
    "txcount": 0,
    "keypoololdest": 1623329789,
    "keypoolsize": 1000,
    "hdseedid": "x",
    "keypoolsize_hd_internal": 1000,
    "paytxfee": 0,
    "private_keys_enabled": true,
    "avoid_reuse": false,
    "scanning": false,
    "descriptors": false
  },
  "error": null,
  "id": "tutorial"
}

Create another wallet, named test-wallet:

$ curl -u "$bitcoinauth" -d '{"jsonrpc": "1.0", "id": "curltest", "method": "createwallet", "params": ["test-wallet"]}' -H 'content-type: text/plain;' http://127.0.0.1:18332/
{"result":{"name":"test-wallet","warning":""},"error":null,"id":"curltest"}

After we created our wallet, list the wallets again:

$ curl -u "$bitcoinauth" -d '{"jsonrpc": "1.0", "id": "curltest", "method": "listwallets", "params": []}' -H 'content-type: text/plain;' http://127.0.0.1:18332/
{"result":["wallet","test-wallet"],"error":null,"id":"curltest"}

Now that we have 2 wallets, we need to specify the wallet name, when we want to do a getwalletinfo method for a specific wallet, test-wallet in this case:

$ curl -s -u "$bitcoinauth" -d '{"jsonrpc": "1.0", "id": "curltest", "method": "getwalletinfo", "params": []}' -H 'content-type: text/plain;' http://127.0.0.1:18332/wallet/test-wallet | jq .
{
  "result": {
    "walletname": "test-wallet",
    "walletversion": 169900,
    "format": "bdb",
    "balance": 0,
    "unconfirmed_balance": 0,
    "immature_balance": 0,
    "txcount": 0,
    "keypoololdest": 1623333495,
    "keypoolsize": 1000,
    "hdseedid": "x",
    "keypoolsize_hd_internal": 1000,
    "paytxfee": 0,
    "private_keys_enabled": true,
    "avoid_reuse": false,
    "scanning": false,
    "descriptors": false
  },
  "error": null,
  "id": "curltest"
}

In order to understand where the data resides for our test-wallet, we can use find:

$ find /blockchain/ -type f -name wallet.dat | grep test-wallet
/blockchain/bitcoin/data/testnet3/wallets/test-wallet/wallet.dat

$ find /blockchain/ -type f -name db.log | grep test-wallet
/blockchain/bitcoin/data/testnet3/wallets/test-wallet/db.log

Include the wallet name in the config located at ~/.bitcoin/bitcoin.conf:

[test]
wallet=wallet
wallet=test-wallet

Then restart bitcoind:

$ sudo systemctl restart bitcoind

Now list our wallets again, and you should see they are being read from config and the wallets will persist if your node restarts:

$ curl -u "$bitcoinauth" -d '{"jsonrpc": "1.0", "id": "curltest", "method": "listwallets", "params": []}' -H 'content-type: text/plain;' http://127.0.0.1:18332/
{"result":["wallet","test-wallet"],"error":null,"id":"curltest"}

To backup a wallet, the test-wallet in this case:

$ curl -u "$bitcoinauth" -d '{"jsonrpc": "1.0", "id": "curltest", "method": "backupwallet", "params": ["test-wallet_bak.dat"]}' -H 'content-type: text/plain;' http://127.0.0.1:18332/wallet/test-wallet
{"result":null,"error":null,"id":"curltest"}

To check where the file was backed up:

$ find /blockchain/ -name test-wallet_bak.dat
/blockchain/bitcoin/data/test-wallet_bak.dat

Addresses

At this moment we have wallets, but we don't have any addresses associated to those wallets, we can verify this by listing wallet addresses using the getaddressesbylabel and passing a empty label as new addresses gets no labels assigned by default.

$ curl -u "$bitcoinauth" -d '{"jsonrpc": "1.0", "id": "curltest", "method": "getaddressesbylabel", "params": [""]}' -H 'content-type: text/plain;' http://127.0.0.1:18332/wallet/test-wallet
{"result":{"":{"purpose":"receive"}},"error":null,"id":"curltest"}

Let's generate a new address for our test-wallet:

$ curl -u "$bitcoinauth" -d '{"jsonrpc": "1.0", "id": "curltest", "method": "getnewaddress", "params": []}' -H 'content-type: text/plain;' http://127.0.0.1:18332/wallet/test-wallet
{"result":"tb1qr66hw59k958xrz008n679p8r9n2y7mjfr3tsjc","error":null,"id":"curltest"}

As you can see our address for test-wallet is tb1qr66hw59k958xrz008n679p8r9n2y7mjfr3tsjc, note that you can have multiple addresses per wallet.

To get address information for the wallet by using the getaddressinfo method and passing the wallet address as the parameter:

$ curl -s -u "$bitcoinauth" -d '{"jsonrpc": "1.0", "id": "curltest", "method": "getaddressinfo", "params": ["tb1qr66hw59k958xrz008n679p8r9n2y7mjfr3tsjc"]}' -H 'content-type: text/plain;' http://127.0.0.1:18332/wallet/test-wallet | jq .
{
  "result": {
    "address": "tb1qr66hw59k958xrz008n679p8r9n2y7mjfr3tsjc",
    "scriptPubKey": "x",
    "ismine": true,
    "solvable": true,
    "desc": "wpkh([05c34822/0'/0'/0']x)#x3x5vu3t",
    "iswatchonly": false,
    "isscript": false,
    "iswitness": true,
    "witness_version": 0,
    "witness_program": "1eb57750b62dxe284e32cd44f6e49",
    "pubkey": "023a1250c0d44751b604656x649357b5e530b9f8500f03ab5b",
    "ischange": false,
    "timestamp": 1623333494,
    "hdkeypath": "m/0'/0'/0'",
    "hdseedid": "x",
    "hdmasterfingerprint": "x",
    "labels": [
      ""
    ]
  },
  "error": null,
  "id": "curltest"
}

As before, we can view the address by label, to view the address for your wallet, we will now see our address:

$ curl -u "$bitcoinauth" -d '{"jsonrpc": "1.0", "id": "curltest", "method": "getaddressesbylabel", "params": [""]}' -H 'content-type: text/plain;' http://127.0.0.1:18332/wallet/test-wallet
{"result":{"tb1qr66hw59k958xrz008n679p8r9n2y7mjfr3tsjc":{"purpose":"receive"}},"error":null,"id":"curltest"}

Get available wallet balance with at least 6 confirmations,:

$ curl -u "$bitcoinauth" -d '{"jsonrpc": "1.0", "id": "curltest", "method": "getbalance", "params": ["*", 6]}' -H 'content-type: text/plain;' http://127.0.0.1:18332/wallet/test-wallet
{"result":0.00000000,"error":null,"id":"curltest"}

Get balances (all balances) for the test-wallet wallet:

$ curl -u "$bitcoinauth" -d '{"jsonrpc": "1.0", "id": "curltest", "method": "getbalances", "params": []}' -H 'content-type: text/plain;' http://127.0.0.1:18332/wallet/test-wallet
{"result":{"mine":{"trusted":0.00000000,"untrusted_pending":0.00000000,"immature":0.00000000}},"error":null,"id":"curltest"}

Create a new address:

$ curl -u "$bitcoinauth" -d '{"jsonrpc": "1.0", "id": "curltest", "method": "getnewaddress", "params": []}' -H 'content-type: text/plain;' http://127.0.0.1:18332/wallet/test-wallet
{"result":"tb1qa7e0mmgsul6pnxhzx7rw49y9qf35enqqra47hh","error":null,"id":"curltest"}

List all addresses for the wallet:

# by default new addresses has no labels, therefore it returns both
$ curl -u "$bitcoinauth" -d '{"jsonrpc": "1.0", "id": "curltest", "method": "getaddressesbylabel", "params": [""]}' -H 'content-type: text/plain;' http://127.0.0.1:18332/wallet/test-wallet
{"result":{"tb1qr66hw59k958xrz008n679p8r9n2y7mjfr3tsjc":{"purpose":"receive"},"tb1qa7e0mmgsul6pnxhzx7rw49y9qf35enqqra47hh":{"purpose":"receive"}},"error":null,"id":"curltest"}

Labelling Addresses

Now we can label addresses on wallets, to label the first address as "green":

$ curl -u "$bitcoinauth" -d '{"jsonrpc": "1.0", "id": "curltest", "method": "setlabel", "params": ["tb1qr66hw59k958xrz008n679p8r9n2y7mjfr3tsjc", "green"]}' -H 'content-type: text/plain;' http://127.0.0.1:18332/wallet/test-wallet
{"result":null,"error":null,"id":"curltest"}

Label the new address as "blue":

$ curl -u "$bitcoinauth" -d '{"jsonrpc": "1.0", "id": "curltest", "method": "setlabel", "params": ["tb1qa7e0mmgsul6pnxhzx7rw49y9qf35enqqra47hh", "blue"]}' -H 'content-type: text/plain;' http://127.0.0.1:18332/wallet/test-wallet
{"result":null,"error":null,"id":"curltest"}

Now we can list addresses for our wallet by the label, "blue" in this example:

$ curl -u "$bitcoinauth" -d '{"jsonrpc": "1.0", "id": "curltest", "method": "getaddressesbylabel", "params": ["blue"]}' -H 'content-type: text/plain;' http://127.0.0.1:18332/wallet/test-wallet
{"result":{"tb1qa7e0mmgsul6pnxhzx7rw49y9qf35enqqra47hh":{"purpose":"receive"}},"error":null,"id":"curltest"}

List addresses for our wallet by the label "green":

$ curl -u "$bitcoinauth" -d '{"jsonrpc": "1.0", "id": "curltest", "method": "getaddressesbylabel", "params": ["green"]}' -H 'content-type: text/plain;' http://127.0.0.1:18332/wallet/test-wallet
{"result":{"tb1qr66hw59k958xrz008n679p8r9n2y7mjfr3tsjc":{"purpose":"receive"}},"error":null,"id":"curltest"}

Create another address for our test-walet:

$ curl -u "$bitcoinauth" -d '{"jsonrpc": "1.0", "id": "curltest", "method": "getnewaddress", "params": []}' -H 'content-type: text/plain;' http://127.0.0.1:18332/wallet/test-wallet
{"result":"tb1qunk223dztk2j2zqswleyenwu3chfqt642vrp8z","error":null,"id":"curltest"}

Set the new address to the green label:

$ curl -u "$bitcoinauth" -d '{"jsonrpc": "1.0", "id": "curltest", "method": "setlabel", "params": ["tb1qunk223dztk2j2zqswleyenwu3chfqt642vrp8z", "green"]}' -H 'content-type: text/plain;' http://127.0.0.1:18332/wallet/test-wallet
{"result":null,"error":null,"id":"curltest"}

List addresses by green label:

$ curl -u "$bitcoinauth" -d '{"jsonrpc": "1.0", "id": "curltest", "method": "getaddressesbylabel", "params": ["green"]}' -H 'content-type: text/plain;' http://127.0.0.1:18332/wallet/test-wallet
{"result":{"tb1qr66hw59k958xrz008n679p8r9n2y7mjfr3tsjc":{"purpose":"receive"},"tb1qunk223dztk2j2zqswleyenwu3chfqt642vrp8z":{"purpose":"receive"}},"error":null,"id":"curltest"}

Receive tBTC

You can receive free test btc, by using any of these testnet faucet websites to receive tBTC over testnet:

The transaction details for sending 0.001 tbtc to my tb1qr66hw59k958xrz008n679p8r9n2y7mjfr3tsjc address, we will receive the following information:

TxID: 637ea98aca23411059ad79aca7ea36ae30b68a173d89e6644703a06a1a846c25
Destination Address: tb1qr66hw59k958xrz008n679p8r9n2y7mjfr3tsjc
Amount: 0.001

Then to list transactions:

$ curl -s -u "$bitcoinauth" -d '{"jsonrpc": "1.0", "id": "curltest", "method": "listtransactions", "params": ["*"]}' -H 'content-type: text/plain;' http://127.0.0.1:18332/wallet/test-wallet | jq .
{
  "result": [
    {
      "address": "tb1qr66hw59k958xrz008n679p8r9n2y7mjfr3tsjc",
      "category": "receive",
      "amount": 0.001,
      "label": "green",
      "vout": 1,
      "confirmations": 0,
      "trusted": false,
      "txid": "637ea98aca23411059ad79aca7ea36ae30b68a173d89e6644703a06a1a846c25",
      "walletconflicts": [],
      "time": 1623337058,
      "timereceived": 1623337058,
      "bip125-replaceable": "no"
    }
  ],
  "error": null,
  "id": "curltest"
}

As you can see at the time, there were 0 confirmations, we can see the same txid as well as other info. With the testnet, we require at least 1 confirmation before a transaction is confirmed, where the mainnet requires 6.

To get balances for our wallet:

$ curl -u "$bitcoinauth" -d '{"jsonrpc": "1.0", "id": "curltest", "method": "getbalances", "params": []}' -H 'content-type: text/plain;' http://127.0.0.1:18332/wallet/test-wallet
{"result":{"mine":{"trusted":0.00000000,"untrusted_pending":0.00100000,"immature":0.00000000}},"error":null,"id":"curltest"}

As you can see as we don't have any confirmations yet, so therefore the trusted value is still 0.

Listing the transactions over time:

$ curl -s -u "$bitcoinauth" -d '{"jsonrpc": "1.0", "id": "curltest", "method": "listtransactions", "params": ["*"]}' -H 'content-type: text/plain;' http://127.0.0.1:18332/wallet/test-wallet | jq .
{
  "result": [
    {
      "address": "tb1qr66hw59k958xrz008n679p8r9n2y7mjfr3tsjc",
      "category": "receive",
      "amount": 0.001,
      "label": "green",
      "vout": 1,
      "confirmations": 0,
      "trusted": false,
      "txid": "637ea98aca23411059ad79aca7ea36ae30b68a173d89e6644703a06a1a846c25",
      "walletconflicts": [],
      "time": 1623337058,
      "timereceived": 1623337058,
      "bip125-replaceable": "no"
    }
  ],
  "error": null,
  "id": "curltest"
}

After a couple of minutes:

$ curl -s -u "$bitcoinauth" -d '{"jsonrpc": "1.0", "id": "curltest", "method": "listtransactions", "params": ["*"]}' -H 'content-type: text/plain;' http://127.0.0.1:18332/wallet/test-wallet | jq .
{
  "result": [
    {
      "address": "tb1qr66hw59k958xrz008n679p8r9n2y7mjfr3tsjc",
      "category": "receive",
      "amount": 0.001,
      "label": "green",
      "vout": 1,
      "confirmations": 2,
      "blockhash": "0000000000000000ba226ad21b51fe3998180dc354ec433ad7a4c4897e04d805",
      "blockheight": 2004280,
      "blockindex": 107,
      "blocktime": 1623337883,
      "txid": "637ea98aca23411059ad79aca7ea36ae30b68a173d89e6644703a06a1a846c25",
      "walletconflicts": [],
      "time": 1623337058,
      "timereceived": 1623337058,
      "bip125-replaceable": "no"
    }
  ],
  "error": null,
  "id": "curltest"
}

Block Explorer

We can also use a blockchain explorer, head over to a testnet blockchain explorer, such as:

https://blockstream.info/testnet/tx

And provide the txid, in my case it was this one:

https://blockstream.info/testnet/tx/637ea98aca23411059ad79aca7ea36ae30b68a173d89e6644703a06a1a846c25

This transaction was done a while ago, so the confirmations will be much more than from the output above, but you can see the confirmations, addresses involved and tbtc amount.

To only get the trusted balance, using the getbalance method and with at least 6 confirmations:

$ curl -u "$bitcoinauth" -d '{"jsonrpc": "1.0", "id": "curltest", "method": "getbalance", "params": ["*", 6]}' -H 'content-type: text/plain;' http://127.0.0.1:18332/wallet/test-wallet
{"result":0.00100000,"error":null,"id":"curltest"}

Let's send another transaction, the list the transactions using the listtransactions method:

$ curl -s -u "$bitcoinauth" -d '{"jsonrpc": "1.0", "id": "curltest", "method": "listtransactions", "params": ["*"]}' -H 'content-type: text/plain;' http://127.0.0.1:18332/wallet/test-wallet | jq .
{
  "result": [
    {
      "address": "tb1qr66hw59k958xrz008n679p8r9n2y7mjfr3tsjc",
      "category": "receive",
      "amount": 0.001,
      "label": "green",
      "vout": 1,
      "confirmations": 4,
      "blockhash": "0000000000000000ba226ad21b51fe3998180dc354ec433ad7a4c4897e04d805",
      "blockheight": 2004280,
      "blockindex": 107,
      "blocktime": 1623337883,
      "txid": "637ea98aca23411059ad79aca7ea36ae30b68a173d89e6644703a06a1a846c25",
      "walletconflicts": [],
      "time": 1623337058,
      "timereceived": 1623337058,
      "bip125-replaceable": "no"
    },
    {
      "address": "tb1qr66hw59k958xrz008n679p8r9n2y7mjfr3tsjc",
      "category": "receive",
      "amount": 0.0111048,
      "label": "green",
      "vout": 0,
      "confirmations": 1,
      "blockhash": "000000000000002912e2da87e6e752c38965fc21e108aab439fcdcd82ba6e37a",
      "blockheight": 2004283,
      "blockindex": 4,
      "blocktime": 1623338496,
      "txid": "3cac023b088a2ddb2d601538edfc72cd1bff1bd2e1a1531518500c5b7a52e473",
      "walletconflicts": [],
      "time": 1623338453,
      "timereceived": 1623338453,
      "bip125-replaceable": "no"
    }
  ],
  "error": null,
  "id": "curltest"
}

After 12 hours, we can see that we have 102 confirmations for our first transaction and 99 transactions for the second transaction:

$ curl -u "$bitcoinauth" -d '{"jsonrpc": "1.0", "id": "curltest", "method": "listtransactions", "params": ["*"]}' -H 'content-type: text/plain;' http://127.0.0.1:18332/wallet/test-wallet | jq .
{
  "result": [
    {
      "address": "tb1qr66hw59k958xrz008n679p8r9n2y7mjfr3tsjc",
      "category": "receive",
      "amount": 0.001,
      "label": "green",
      "vout": 1,
      "confirmations": 102,
      "blockhash": "0000000000000000ba226ad21b51fe3998180dc354ec433ad7a4c4897e04d805",
      "blockheight": 2004280,
      "blockindex": 107,
      "blocktime": 1623337883,
      "txid": "637ea98aca23411059ad79aca7ea36ae30b68a173d89e6644703a06a1a846c25",
      "walletconflicts": [],
      "time": 1623337058,
      "timereceived": 1623337058,
      "bip125-replaceable": "no"
    },
    {
      "address": "tb1qr66hw59k958xrz008n679p8r9n2y7mjfr3tsjc",
      "category": "receive",
      "amount": 0.0111048,
      "label": "green",
      "vout": 0,
      "confirmations": 99,
      "blockhash": "000000000000002912e2da87e6e752c38965fc21e108aab439fcdcd82ba6e37a",
      "blockheight": 2004283,
      "blockindex": 4,
      "blocktime": 1623338496,
      "txid": "3cac023b088a2ddb2d601538edfc72cd1bff1bd2e1a1531518500c5b7a52e473",
      "walletconflicts": [],
      "time": 1623338453,
      "timereceived": 1623338453,
      "bip125-replaceable": "no"
    },
    {
      "address": "tb1qr66hw59k958xrz008n679p8r9n2y7mjfr3tsjc",
      "category": "receive",
      "amount": 0.03521065,
      "label": "green",
      "vout": 5,
      "confirmations": 27,
      "blockhash": "000000000000004255a9d5af67b4649ff3f4d6a2f0c334261ca822cd9fbd00a9",
      "blockheight": 2004355,
      "blockindex": 43,
      "blocktime": 1623383177,
      "txid": "eb43868bd2c5abd97d4f5f11450952837bc3edc149478248e9453fdfb05c5187",
      "walletconflicts": [],
      "time": 1623382990,
      "timereceived": 1623382990,
      "bip125-replaceable": "no"
    }
  ],
  "error": null,
  "id": "curltest"
}

After 3 transactions, view the balance in the test wallet using the getbalance method:

$ curl -u "$bitcoinauth" -d '{"jsonrpc": "1.0", "id": "curltest", "method": "getbalance", "params": ["*", 6]}' -H 'content-type: text/plain;' http://127.0.0.1:18332/wallet/test-wallet
{"result":0.04731545,"error":null,"id":"curltest"}

Sending a raw transaction

A easier way to send a transaction is by using sendtoaddress and the source wallet will be in the request url, ie: /wallet/wallet-name

First we look if we have a address for our account that we are sending from, if not then we can create a wallet and a address, for this example, I have a address for the wallet wallet:

$ curl -s -u "$bitcoinauth" -d '{"jsonrpc": "1.0", "id": "curl", "method": "getaddressesbylabel", "params": [""]}' -H 'content-type: text/plain;' http://127.0.0.1:18332/wallet/wallet
{"result":{"tb1qks4tyrz52vvdh35kcx0ypvnj3fjdkl692pzfyc":{"purpose":"receive"}},"error":null,"id":"curl"}

Ensure that our source wallet has funds in it:

$ curl -s -u "$bitcoinauth" -d '{"jsonrpc": "1.0", "id": "curl", "method": "getbalance", "params": []}' -H 'content-type: text/plain;' http://127.0.0.1:18332/wallet/wallet | python -m json.tool
{
    "error": null,
    "id": "curl",
    "result": 0.01811929
}

We have enough funds to send, so we now have the source wallet name and we need to get the wallet address, where we want to send the funds to, which in this case is the address in test-wallet as the destination:

curl -s -u "$bitcoinauth" -d '{"jsonrpc": "1.0", "id": "curl", "method": "getaddressesbylabel", "params": [""]}' -H 'content-type: text/plain;' http://127.0.0.1:18332/wallet/test-wallet
{"result":{"tb1qzxmefmcpq98z42v67a80gvug2fe979r5h768yv":{"purpose":"receive"}},"error":null,"id":"curl"}

Just to double check our current funds in the wallet that will receive funds:

$ curl -s -u "$bitcoinauth" -d '{"jsonrpc": "1.0", "id": "curl", "method": "getbalance", "params": []}' -H 'content-type: text/plain;' http://127.0.0.1:18332/wallet/test-wallet | python -m json.tool
{
    "error": null,
    "id": "curl",
    "result": 0.35572584
}

Now we will use the sendtoaddress method, with the recipient address and the amount to send as the parameters. To summarize:

Source Wallet: wallet
Destination Address: tb1qzxmefmcpq98z42v67a80gvug2fe979r5h768yv
Amount to send: 0.01.

Sending the amount:

$ curl -s -u "$bitcoinauth"  -d '{"jsonrpc": "1.0", "id":"0", "method": "sendtoaddress", "params":["tb1qzxmefmcpq98z42v67a80gvug2fe979r5h768yv", 0.01]}' -H 'content-type: text/plain;' http://127.0.0.1:18332/wallet/wallet
{"result":"df087095ac79d678f9d98c8bf8ebface2ac62a20546d85e07a852feb2c3bea50","error":null,"id":"0"}

We will receive a transaction id, and if we list for transactions for our source wallet, we will see the transaction:

$ curl -s -u "$bitcoinauth" -d '{"jsonrpc": "1.0", "id": "curl", "method": "listtransactions", "params": []}' -H 'content-type: text/plain;' http://127.0.0.1:18332/wallet/wallet | python -m json.tool
{
    "error": null,
    "id": "curl",
    "result": [
        {
            "address": "tb1qks4tyrz52vvdh35kcx0ypvnj3fjdkl692pzfyc",
            "amount": 0.01811929,
            "bip125-replaceable": "no",
            "blockhash": "000000000000003c0ac4978ba815ff8f0d7f55da98923c686118c75461fb579e",
            "blockheight": 2091275,
            "blockindex": 1,
            "blocktime": 1630677226,
            "category": "receive",
            "confirmations": 1,
            "label": "",
            "time": 1630677177,
            "timereceived": 1630677177,
            "txid": "e44b45a309284e8044a15dca8c0a895a5c7072741882281038fb185cc0c1a0d9",
            "vout": 0,
            "walletconflicts": []
        },
        {
            "abandoned": false,
            "address": "tb1qzxmefmcpq98z42v67a80gvug2fe979r5h768yv",
            "amount": -0.01,
            "bip125-replaceable": "no",
            "category": "send",
            "confirmations": 0,
            "fee": -1.41e-06,
            "time": 1630677743,
            "timereceived": 1630677743,
            "trusted": true,
            "txid": "df087095ac79d678f9d98c8bf8ebface2ac62a20546d85e07a852feb2c3bea50",
            "vout": 0,
            "walletconflicts": []
        }
    ]
}

So when we look at the sender wallet, we will see the funds was deducted:

$ curl -s -u "$bitcoinauth" -d '{"jsonrpc": "1.0", "id": "curl", "method": "getbalance", "params": []}' -H 'content-type: text/plain;' http://127.0.0.1:18332/wallet/wallet | python -m json.tool
{
    "error": null,
    "id": "curl",
    "result": 0.00811788
}

And when we look at the receiver wallet, we can see that the account was received:

$ curl -s -u "$bitcoinauth" -d '{"jsonrpc": "1.0", "id": "curl", "method": "getbalance", "params": []}' -H 'content-type: text/plain;' http://127.0.0.1:18332/wallet/rpi01-main | python -m json.tool
{
    "error": null,
    "id": "curl",
    "result": 0.36572584
}

Thank You

Thanks for reading, if you like my content, check out my website or follow me at @ruanbekker on Twitter.

The source code of this blog post will be added to this Github Repository:

https://github.com/ruanbekker/bitcoin-full-node-demo

Running a Testnet with Bitcoin on Linux

Ruan Bekker — Mon, 13 Sep 2021 07:14:02 +0000

This will be a series of posts on:

The basics of Blockchain (this post)
Setting up a Bitcoin Full-Node on Linux (this post)
CLI and JSON RPC examples to interact with the blockchain and your wallet (post 2)

In this in-depth tutorial, we will cover the basics of blockchain, why you would want a bitcoin full-node, how to setup a bitcoin full-node on linux using the testnet chain and how to interact with your node and the blockchain using the cli and the json rpc, where we will create wallets and addresses and sending tbtc to your accounts.

Blockchain Basics

Before we start setting up our bitcoin full-node, we first need to get through some blockchain basics, if you already aware of it, you can skip the the setup section of this post.

Block

Transaction data is permanently recorded into files called blocks. You can think of it as a transaction ledger. Blocks are organised into a linear sequence over time.

New transactions are constantly being processed by miners into new blocks which are added to the end of the chain. As blocks are buried deeper and deeper into the blockchain they become harder and harder to change or remove, this gives rise of Bitcoin's Irreversible Transactions.

The first block added to the blockchain is referred to as the genesis block

Blockchain

A blockchain is a transaction database shared by all nodes participating in a system based on the bitcoin protocol. A full copy of a currency's blockchain contains every transaction ever executed in the currency. With this information, one can find out how much value belonged to each address at any point in history.

Every block contains a hash of the previous block. This has the effect of creating a chain of blocks from the genesis block to the current block. Each block is guaranteed to come after the previous block chronologically because the previous block's hash would otherwise not be known. Each block is also computationally impractical to modify once it has been in the chain for a while because every block after it would also have to be regenerated. These properties are what make bitcoins transactions irreversible. The blockchain is the main innovation of Bitcoin.

Mining

Mining is the process of adding transaction records to bitcoin's public ledger of past transactions. The term "mining rig" is referred to where as a single computer system that performs the necessary computations for "mining".

The blockchain serves to confirm transactions to the rest of the network as having taken place. Bitcoin nodes use the blockchain to distinguish legitimate Bitcoin transactions from attempts to re-spend coins that have already been spent elsewhere.

Node

Any computer that connects to the bitcoin network is called a node. Nodes that fully verify all of the rules of bitcoin are called full nodes. The most popular software implementation of full nodes is called bitcoin-core, its releases can be found on their github page

What is a Full Node

A full node is a node (computer system with bitcoin-core running on it) which downloads every block and transaction and check them against bitcoin's consensus rules. which fully validates transactions and blocks. Almost all full nodes also help the network by accepting transactions and blocks from other full nodes, validating those transactions and blocks, and then relaying them to further full nodes.

Some examples of consensus rules:

Blocks may only create a certain number of bitcoins. (Currently 6.25 BTC per block.)
Transactions must have correct signatures for the bitcoins being spent.
Transactions/blocks must be in the correct data format.
Within a single blockchain, a transaction output cannot be double-spent.

At minimum, a full node must download every transaction that has ever taken place, all new transactions, and all block headers. Additionally, full nodes must store information about every unspent transaction output until it is spent.

By default full nodes are inefficient in that they download each new transaction at least twice, and they store the entire block chain (more than 165 GB as of 20180214) forever, even though only the unspent transaction outputs (<2 GB) are required. Performance can improved by enabling -blocksonly mode and enabling pruning

Archival Nodes

A subset of full nodes also accept incoming connections and upload old blocks to other peers on the network. This happens if the software is run with -listen=1 as is default.

Contrary to some popular misconceptions, being an archival node is not necessary to being a full node. If a user's bandwidth is constrained then they can use -listen=0, if their disk space is constrained they can use pruning, all the while still being a fully-validating node that enforces bitcoin's consensus rules and contributing to bitcoin's overall security.

Most information was referenced from this wiki.

Setup Bitcoin Core

Now for the fun part, to setup bitcoin-core on linux, in my case I am using a fresh Ubuntu 20.04 instance and the following commands will be executed as a non-root user.

Environment Variables

Set the following environment variables, the latest version for BITCOIN_VERSION can be retrieved at: https://bitcoincore.org/bin/, execute the following:

$ export ARCH=x86_64
$ export BITCOIN_VERSION=0.21.1
$ export BITCOIN_URL=https://bitcoincore.org/bin/bitcoin-core-${BITCOIN_VERSION}/bitcoin-${BITCOIN_VERSION}-${ARCH}-linux-gnu.tar.gz
$ export BITCOIN_DATA_DIR=/blockchain/bitcoin/data

Installation

Create the user and group for bitcoin:

$ sudo groupadd -r bitcoin
$ sudo useradd -r -m -g bitcoin -s /bin/bash bitcoin

Update the package manager and install the dependencies:

$ sudo apt update && sudo apt install ca-certificates gnupg gpg wget jq --no-install-recommends -y

Download bitcoin-core and verify that the package matches the sha hash:

$ cd /tmp
$ wget https://bitcoincore.org/bin/bitcoin-core-${BITCOIN_VERSION}/SHA256SUMS.asc
$ wget -qO bitcoin-${BITCOIN_VERSION}-${ARCH}-linux-gnu.tar.gz "${BITCOIN_URL}"
$ cat SHA256SUMS.asc | grep bitcoin-${BITCOIN_VERSION}-${ARCH}-linux-gnu.tar.gz | awk '{ print $1 }'

Extract the package to /opt/bitcoin/${BITCOIN_VERSION} and exclude any graphical user interfacing binaries, create the home directory and set the ownership:

$ sudo mkdir -p /opt/bitcoin/${BITCOIN_VERSION}
$ sudo mkdir -p ${BITCOIN_DATA_DIR}
$ sudo tar -xzvf bitcoin-${BITCOIN_VERSION}-${ARCH}-linux-gnu.tar.gz -C /opt/bitcoin/${BITCOIN_VERSION} --strip-components=1 --exclude=*-qt
$ sudo ln -s /opt/bitcoin/${BITCOIN_VERSION} /opt/bitcoin/current
$ sudo rm -rf /tmp/*
$ sudo chown -R bitcoin:bitcoin ${BITCOIN_DATA_DIR}

When you have to upgrade the software version of bitcoin-core in the future you can remove the symlink with sudo rm -rf /usr/local/bitcoin/current and symlink to the newer version as shown above.

Configuration

Create the bitcoin configuration, here you see I am using the testnet chain and due to storage restrictions for my use-case I am setting pruning mode to 1GB, and if you don't set BITCOIN_RPC_USER it will use the user bitcoin and if you don't set BITCOIN_RPC_PASSWORD it will generate a password for the json-rpc interface:

$ cat > bitcoin.conf.tmp << EOF
datadir=${BITCOIN_DATA_DIR}
printtoconsole=1
rpcallowip=127.0.0.1
rpcuser=${BITCOIN_RPC_USER:-bitcoin}
rpcpassword=${BITCOIN_RPC_PASSWORD:-$(openssl rand -hex 24)}
testnet=1
prune=1000
[test]
rpcbind=127.0.0.1
rpcport=18332
EOF

Create the systemd unit-file for bitcoind:

$ cat > bitcoind.service <<  EOF
[Unit]
Description=Bitcoin Core Testnet
After=network.target

[Service]
User=bitcoin
Group=bitcoin
WorkingDirectory=${BITCOIN_DATA_DIR}

Type=simple
ExecStart=/usr/local/bitcoin/current/bin/bitcoind -conf=$BITCOIN_DATA_DIR/bitcoin.conf

[Install]
WantedBy=multi-user.target
EOF

Now move the temporary config files, change the ownership and symlink the bitcoin home directory to the path that we created earlier:

$ sudo mv bitcoin.conf.tmp $BITCOIN_DATA_DIR/bitcoin.conf
$ sudo chown bitcoin:bitcoin $BITCOIN_DATA_DIR/bitcoin.conf
$ sudo chown -R bitcoin $BITCOIN_DATA_DIR
$ sudo ln -sfn $BITCOIN_DATA_DIR /home/bitcoin/.bitcoin
$ sudo chown -h bitcoin:bitcoin /home/bitcoin
$ sudo chown -R bitcoin:bitcoin /home/bitcoin

Move the systemd unit file in place, then reload systemd and start bitcoind:

$ sudo mv bitcoind.service /etc/systemd/system/bitcoind.service
$ sudo systemctl daemon-reload
$ sudo systemctl enable bitcoind
$ sudo systemctl start bitcoind

Switch to the bitcoin user:

$ sudo su - bitcoin

And append the bitcoin path to the ~/.profile file so that your user know where to find the bitcoin binaries:

$ export PATH=$PATH:/opt/bitcoin/current/bin

Initial Block Download

Once bitcoind process has started, the initial block download will start and you can get the progress as the bitcoin user using the cli:

$ bitcoin-cli -getinfo

Or as the root user you can view the progress using journalctl:

$ sudo journalctl -fu bitcoind
Sep 03 08:44:23 rpi-01 bitcoind[532]: 2021-09-03T06:44:23Z UpdateTip: new best=x height=2091205 version=0x20c00000 log2_work=74.461678 tx=60938712 date='2021-09-03T06:44:13Z' progress=1.000000 cache=1.7MiB(13673txo)

Troubleshooting

If you run into any issues you can see the status of bitcoind using:

$ sudo systemctl status bitcoind

Or check the logs:

$ sudo journalctl -fu bitcoind

Interacting with the Blockchain

In my next post I will demonstrate how to create a wallet, send some test bitcoin to your wallet and how to track transactions on a public block explorer.

Thank You

Thanks for reading, if you like my content, check out my website or follow me at @ruanbekker on Twitter.

The source code of this blog post will be added to this Github Repository:

https://github.com/ruanbekker/bitcoin-full-node-demo

Why you should use Multi Stage Docker Builds

Ruan Bekker — Sat, 07 Aug 2021 01:43:37 +0000

In this tutorial I will demonstrate how to build slim docker images using multistage docker builds, where you can save up to 800MB of disk space per image.

Originally posted on containers.fan

About

We will use multistage docker builds from a alpine image as our build image to get the dependencies, build the go binary and use our scratch image to place the built binary onto our target image to have small docker images.

Why does size matter

So let's assume you have a orchestrator such as ECS, Swarm or Kubernetes with 100 nodes behind a Auto Scaling Group, where your cluster node count is 10 when theres low traffic, and 100 nodes when theres lots of traffic.

As we scale out our service might scale from 10 replicas to 500 replicas and let's assume our container image is 800MB in size, when a new node joins a cluster the container image won't be in the cache so the docker daemon needs to download that image from the docker registry. So let's say 100 nodes join the cluster and our service scales to 100 replicas, it means that each node needs to download 800MB from the docker registry, that is about 80GB of incoming network throughput to the cluster.

So when we use multistage builds and in this case using Go, we can slim down our container image to less than 3MB, if we do the same calculation, that is just less than 300MB of throughput and if a fresh node joins a cluster and the container image is not present, it will take about a second or two to download and getting the container to run (depending on internet speed) and you obviously save disk space.

Go Application

We will use a library that generates random data from go-randomdata in our application, app.go:

package main

import (
    "fmt"
    "github.com/Pallinder/go-randomdata"
)

func main() {
    profile := randomdata.GenerateProfile(randomdata.Male | randomdata.Female | randomdata.RandomGender)
    fmt.Printf("The new profile's username is: %s and password (md5): %s\n", profile.Login.Username, profile.Login.Md5)
}

Single Stage Docker Build

In this example we will use the golang image to get the dependencies and build the application in one image, our Dockerfile.single_stage:

FROM golang:latest as builder
RUN mkdir -p /go/src/github.com/ruanbekker
WORKDIR /go/src/github.com/ruanbekker
RUN useradd -u 10001 app
COPY . .
ENV GO111MODULE=auto
RUN go get
RUN CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -o main .
USER app
CMD ["/go/src/github.com/ruanbekker/main"]

Building the image:

docker build -f Dockerfile.single_stage -t goapp:singlestage .

Multi Stage Docker Build

Our multi stage consist of a build image where we will use the golang image to fetch our dependencies and build our application, then we use the scratch image as the target to copy the compiled binary to and run the container from the slim image, our Dockerfile.multi_stage:

FROM golang:latest as builder
RUN mkdir -p /go/src/github.com/ruanbekker
WORKDIR /go/src/github.com/ruanbekker
RUN useradd -u 10001 app
COPY . .
ENV GO111MODULE=auto
RUN go get
RUN CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -o main .

FROM scratch
COPY --from=builder /go/src/github.com/ruanbekker/main /main
COPY --from=builder /etc/passwd /etc/passwd
USER app
CMD ["/main"]

Building the image:

docker build -f Dockerfile.multi_stage -t goapp:multistage .

Comparing the size differences

When we compare the size differences of our docker images between a normal build and a multi-stage build we can see a huge difference:

docker images | head -3
REPOSITORY                        TAG                 IMAGE ID            CREATED             SIZE
goapp                             singlestage         0d0c1f4c98a2        54 seconds ago      896MB
goapp                             multistage          d74ac27a39c8        2 hours ago         2.75MB

And just to show that both containers run from the built docker images, our single build:

docker run -it goapp:singlestage
The new profile's username is: Maregrass and password (md5): 56da7705b7648a38f539b043e6a494be

And our multi-stage build:

docker run -it goapp:multistage
The new profile's username is: Shirtplaid and password (md5): 7d8606ee86f2da3ed12c595ab617bf4e

Thank You

If you liked this content, please make sure to share or come say hi on my website or twitter:

Website: ruan.dev
Twitter: @ruanbekker

Basic Logging with Python

Ruan Bekker — Sat, 07 Aug 2021 01:38:47 +0000

I'm trying to force myself to move away from using the print() function as I'm pretty much using print all the time to cater for logging, and using the logging package instead.

This is a basic example of using logging in a basic python app:

import logging

logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s [%(levelname)s] %(name)s %(message)s",
    handlers=[
        logging.StreamHandler()
    ]
)

messagestring = {'info': 'info message', 'warn': 'this is a warning', 'err': 'this is a error'}

logger = logging.getLogger('thisapp')
logger.info('message: {}'.format(messagestring['info']))
logger.warning('message: {}'.format(messagestring['warn']))
logger.error('message: {}'.format(messagestring['err']))

When running this example, this is the output that you will see:

$ python app.py
2021-07-19 13:07:43,647 [INFO] thisapp message: info message
2021-07-19 13:07:43,647 [WARNING] thisapp message: this is a warning
2021-07-19 13:07:43,647 [ERROR] thisapp message: this is a error

More more info on this package, see it's documentation:

https://docs.python.org/3/library/logging.html

Thanks for reading, if you like my content, check out my website or follow me at @ruanbekker on Twitter.