DEV Community: Caio Campos Borges Rosa

How to automate provisioning in Proxmox Using Cloud images

Caio Campos Borges Rosa — Sun, 16 Jun 2024 23:24:56 +0000

There are many ways to create virtual machines. In this article, we aim to leverage cloud images as our primary method for automation while provisioning on Proxmox.

Cloud Images

Cloud images are pre-configured disk images designed to be used in virtualized environments, such as cloud infrastructure or virtual machine hosts. These images include minimal operating system installation and are optimized for quick deployment and scalability. They have support for quick configuration tools, we will be using cloud-init for our example.

Script

You'll need a Proxmox Virtual Environment node with SSH access. In our script, we will perform some actions to download and prepare a cloud image.

Variable and packages First, we set up some variables and install some useful packages we will be using in the script.



### variables
VM_TEMPLATE_ID=999
TEMPLATE_NAME='ubuntu-2204-template'
UBUNTU_IMAGE='ubuntu-22.04-server-cloudimg-amd64-disk-kvm.img'
UBUNTU_IMAGE_QCOW2='ubuntu-22.04.qcow2'
USERNAME='cap'
PASSWORD='12345'
MEMORY='4096'
CPUS='2'

apt update -y && apt install nano wget curl libguestfs-tools -y

Idempotency Run a couple of commands to make our script idempotent. We want a quick way of iteration, so we will be deleting the template in case of changes.



# remove old image
rm -rfv ${UBUNTU_IMAGE}

# remove old template container - WILL DESTROY COMPLETELY
qm destroy ${VM_TEMPLATE_ID} --destroy-unreferenced-disks 1 --purge 1

Download the image:



# download new image
wget http://cloud-images.ubuntu.com/releases/22.04/release/${UBUNTU_IMAGE}

Customize the image Add the QEMU guest agent to the image so we don't need to install it later. Here, you can use virt-customize to add other tools and have them pre-installed on the image.



virt-customize -a ${UBUNTU_IMAGE} --install qemu-guest-agent

QEMU guest agent is a daemon installed in the guest and helps us get information about our VMs in the Proxmox environment. See more documentation on PVE docs.

Change image extension Change the img extension on the image file to QCOW2, if you don't rename it in some versions of proxmox qemu will not work



mv ${UBUNTU_IMAGE} ${UBUNTU_IMAGE_QCOW2}

Resize
Now we resize the image to a generic size. It doesn't really matter now; we will configure each clone based on the template later.
Create the vm

Again, the values for resources here are not really important. Set up a generic value for memory and CPU. We are setting up a network interface. Another key configuration is the storage controller; we use virtio-scsi as it covers more use cases. To learn more about virtualization of 'physical' devices on the virtio family and emulated storage controllers of the virtio family, go to Virtio



qm create ${VM_TEMPLATE_ID} --memory ${MEMORY} --cores ${CPUS} --net0 virtio,bridge=vmbr0 --name ${TEMPLATE_NAME} --scsihw virtio-scsi-pci

After creating the vm we need to configure and convert to a template, so it can be cloned.

Configure the vm

We will perform in order the following configuration:

Import the image to a disk and attach to the vm
attach a cloud-init drive to the vm via ide interface
Configuring vga output for the console on serial0
Setting up dhcp for the network so we have connection
enabling the qemu agent
seting default user and password
adding ssh keys so we can have ssh access, it will use the same ssh keys from the host in witch you will be executing the script



qm set ${VM_TEMPLATE_ID} --scsi0 local-lvm:0,import-from=/root/${UBUNTU_IMAGE_QCOW2}
qm set ${VM_TEMPLATE_ID} --ide2 local-lvm:cloudinit
qm set ${VM_TEMPLATE_ID} --boot order=scsi0
qm set ${VM_TEMPLATE_ID} --serial0 socket --vga serial0
qm set ${VM_TEMPLATE_ID} --ipconfig0 ip=dhcp
qm set ${VM_TEMPLATE_ID} --agent enabled=1
qm set ${VM_TEMPLATE_ID} -ciuser ${USERNAME}
qm set ${VM_TEMPLATE_ID} -cipassword ${PASSWORD}
qm set ${VM_TEMPLATE_ID} --sshkeys ~/.ssh/authorized_keys

Converting to template

In the last step, we convert this VM to a template. A template is a VM with a frozen state used as a base for many machines. It provides a stable starting point and makes it easy for provisioning automation.



qm template ${VM_TEMPLATE_ID}

The complete script can be found on my homelab github

To run over ssh we can pipe the output via ssh running:



cat ubuntu-22.04.sh | ssh root@your-PVE-ip /bin/bash

Result should be a template on your Proxmox node

Now you can run any number of copies, we can use qemu set to configure any variable to the clone we want to change from the template.



# clone

qm clone ${VM_TEMPLATE_ID} ${VM_ID} --name ${VM_NAME}

# configure the vm  
qm set ${VM_ID} --scsi1 local-lvm:40
qm set ${VM_ID} --memory ${MEMORY}
qm set ${VM_ID} --cores ${CPUS}
qm set ${VM_ID} --boot order=scsi0
qm set ${VM_ID} --serial0 socket --vga serial0
qm set ${VM_ID} --ipconfig0 ip=dhcp
qm set ${VM_ID} --agent enabled=1

# start
qm start ${VM_ID}

Running again over ssh should create a new vm based on the template:



cat clone-templates.sh | ssh root@your-PVE-ip /bin/bash

Its as fast as it gets, less than 30 seconds and you have a vm running:

Now we know how to create vms based on templates we can explore more options of remote code execution for provisioning.

Socials:

Twitter
Linkedin
Github

Photo by Arno Senoner on Unsplash

How to automate tests with Tekton

Caio Campos Borges Rosa — Mon, 20 May 2024 11:44:07 +0000

Following our series on CI/CD cloud native, we will go on setting up a simple Tekton pipeline to automate testing, using kubernetes. We should cover the simple flow of updating code and testing code. We will be using GitHub webhook events to trigger our pipeline.

First we need to install Tekton operator, so we can focus configuration in one config file, making it easy to have all the features we need in a more declarative way.

Installing Tekton Operator

First, install the Operator Lifecycle Manager, a tool to manage operators running in your cluster.



curl -sL https://github.com/operator-framework/operator-lifecycle-manager/releases/download/v0.27.0/install.sh | bash -s v0.27.0

Install the operator:




kubectl create -f https://operatorhub.io/install/tektoncd-operator.yaml

Wait until the operator is up and running. You can check the status by running:



kubectl get csv -n operators

Configure Tekton

We will need to configure Tekton. The operator is configured using a file called TektonConfig.yaml.



apiVersion: operator.tekton.dev/v1alpha1
kind: TektonConfig
metadata:
  name: config
  namespace: tekton-pipelines
spec:
  targetNamespace: tekton-pipelines
  profile: all
  chain:
    disabled: false
  pipeline:
    await-sidecar-readiness: false
    disable-affinity-assistant: true
    disable-creds-init: false
    enable-api-fields: alpha
    enable-bundles-resolver: true
    enable-cluster-resolver: true
    enable-custom-tasks: true
    enable-git-resolver: true
    performance:
      disable-ha: true
      buckets: 1
      replicas: 1
      threads-per-controller: 32
      kube-api-qps: 100.0
      kube-api-burst: 200
  pruner:
    disabled: false
    schedule: "0 * * * *"
    resources:
      - taskrun
      - pipelinerun
    keep: 3
    # keep-since: 1440
    # NOTE: you can use either keep or keep-since, not both
    prune-per-resource: true
  hub:
    params:
      - name: enable-devconsole-integration
        value: "true"
    options:
      disabled: false
  dashboard:
    readonly: false
    options:
      disabled: false

We will go on about some items in the config file, but you can find the full reference on the TektonConfig.

We are setting up the profile of the operator to all as this gives us access to all the features on the Tekton operator. If you plan to use less and want a slimmer setup, the reference for profiles is:

all: This profile will install all components (TektonPipeline, TektonTrigger, and TektonChain)

basic: This profile will install only TektonPipeline, TektonTrigger, and TektonChain components

lite: This profile will install only TektonPipeline components

We are disabling affinity assistant. This is a feature that coschedules pods of a PipelineRun that share the same persistent volume to the same Node. This is being deprecated in favor of coschedule workspaces. We are also disabling sidecar readiness, as we won't be using any sidecars.

Pruner configuration is configured to run at the beginning of each hour. It will delete Tasks and TaskRuns, clearing resources.

Dashboard readonly false will allow for actions on PipelineRuns and tasks to be taken directly on the dashboard.

Overview

Let's visualize what we want to perform in our CI/CD flow.
First, we will make changes to our codebase. Those changes will be pushed to a GitHub repository. The push event on the GitHub webhook will trigger our pipeline to run tests.

Handling Github Events

For external events, we will set up an event listener. Tekton uses a custom resource for that, EventListener. This will create a service exposed via Kubernetes API. This service will receive GitHub webhook events. Then, after that, we can write a filter to match the event and trigger on our event listener.



---
apiVersion: triggers.tekton.dev/v1beta1
kind: EventListener
metadata:
  name: tekton-github-pr-{{ .Values.projectName }}
spec:
  serviceAccountName: service-account-{{ .Values.projectName }}
  triggers:
    - name: pr-trigger
      interceptors:
        - ref:
            name: "cel"
            kind: ClusterInterceptor
            apiVersion: triggers.tekton.dev
          params:
            - name: "filter"
              value: >
                header.match('x-github-event', 'merge')
            - name: "overlays"
              value:
                - key: author
                  expression: body.pusher.name.lowerAscii().replace('/','-').replace('.', '-').replace('_', '-')
                - key: pr-ref
                  expression: body.ref.lowerAscii().replace("/", '-')
      bindings:
        - ref: tb-github-pr-trigger-binding-{{ .Values.projectName }}
      template:
        ref: tt-github-pr-trigger-template-{{ .Values.projectName }}

Notice this resource has a serviceAccountName as required. The EventListener will create resources in our cluster. This will ensure we have the correct roles and permissions to do it properly.

We will use a Cel ClusterInterceptor, another custom resource so we can write filter expressions using CEL, This is how we manage to evaluate the webhook request and filter triggers for many kinds of pipelines.

Here we also use overlays to create variables based on an expression that can be passed down to our pipelines. In this case, we want the author and the ref so we can customize the pipeline display name.

Bindings and templates are two other resources we will reference in the EventListener.

Bindings

TriggerBindings are another way to bind objects from the webhook request to variables we can use to control pipeline flow.



apiVersion: triggers.tekton.dev/v1beta1
kind: TriggerBinding
metadata:
  name: tb-github-pr-trigger-binding-{{ .Values.projectName }}
spec:
  params:
    - name: revision
      value: $(body.after)
    - name: repo-url
      value: $(body.repository.ssh_url)
    - name: author
      value: $(extensions.author)
    - name: pr-ref
      value: $(extensions.pr-ref)
    - name: repo-full-name
      value: $(body.repository.full_name)

Here we reference from the request captured by the EventListener the information we want to be assigned to variables, so we can pass down to the pipeline in the TriggerBinding. The variables we create with overlays in the EventListener we have to reference from the object extensions. We can reference the request body and header too.

Triggering the Pipeline

TriggerTemplate is the resource that pieces together events with the variables we set up on the TriggerBinding. Here we will associate variables as params to the pipelines, creating a PipelineRun, which is the actual automation being executed as a pod in Kubernetes.



apiVersion: triggers.tekton.dev/v1beta1
kind: TriggerTemplate
metadata:
  name: tt-github-pr-trigger-template-{{ .Values.projectName }}
spec:
  params:
    - name: revision
    - name: repo-url
    - name: author
    - name: repo-full-name
    - name: pr-ref
  resourcetemplates:
    - apiVersion: tekton.dev/v1beta1
      kind: PipelineRun
      metadata:
        generateName: pr-$(tt.params.pr-ref)-$(tt.params.author)-
      spec:
        serviceAccountName: service-account-{{ .Values.projectName }}
        pipelineRef:
          name: {{ .Values.projectName}}-pipeline
        workspaces:
          - name: cache
            persistentVolumeClaim:
              claimName: pvc-cache-{{ .Values.projectName }}
          - name: shared-data
            volumeClaimTemplate:
              spec:
                accessModes:
                  - ReadWriteOnce
                resources:
                  requests:
                    storage: 2Gi
        params:
          - name: repo-url
            value: $(tt.params.repo-url)
          - name: revision
            value: $(tt.params.revision)
          - name: repo-full-name
            value: $(tt.params.repo-full-name)
          - name: ref
            value: $(tt.params.ref)
          - name: deploy-staging
            value: $(tt.params.deploy-staging)
          - name: test-all
            value: $(tt.params.test-all)

On the TriggerTemplate, we receive the params from the TriggerBinding and pass the params to the PipelineRun.
We need to reference the Pipeline on pipelineRef, this is the pipeline we want to run.

We also define our workspaces. In our example, we are passing 2 workspaces, one as a volumeClaimTemplate that will be discarded at the end of the PipelineRun. It uses claim templates to dynamically provision non-persistent storage with the resource values described. The workspace cache is defined using persistentVolumeClaim, which takes a persistent volume you need to define.

Pipeline

The pipeline is the orchestrated flow of tasks we want to run. It will have the parameters we defined in the TriggerTemplate and the logic we want to execute applied to tasks. For this example, we will look at a simple test pipeline that clones a repository and runs the test script.



apiVersion: tekton.dev/v1beta1
kind: Pipeline
metadata:
  name: {{ .Values.projectName }}-pipeline-tests
spec:
  workspaces:
    - name: shared-data
    - name: cache
  params:
    - name: repo-url
      type: string
    - name: revision
      type: string
  tasks:
    - name: fetch-source
      taskRef:
        resolver: cluster
        params:
          - name: kind
            value: task
          - name: name
            value: task-git-clone
          - name: namespace
            value: tekton-pipelines
      params:
        - name: url
          value: $(params.repo-url)
        - name: revision
          value: $(params.revision)
        - name: depth
          value: 2
      workspaces:
        - name: output
          workspace: shared-data
    - name: install-deps
      runAfter: ["fetch-source"]
      taskRef:
        resolver: cluster
        params:
          - name: kind
            value: task
          - name: name
            value: task-install-deps
          - name: namespace
            value: tekton-pipelines
      params: 
        - name: install-script
          value: yarn install --prefer-offline --ignore-engines
      workspaces:
        - name: source
          workspace: shared-data
        - name: cache
          workspace: cache
 - name: test-task
      runAfter: ["install-deps"]
      taskRef:
        resolver: cluster
        params:
          - name: kind
            value: task
          - name: name
            value: task-test
          - name: namespace
            value: tekton-pipelines
      params:
        - name: diff
          value: $(tasks.fetch-source.results.diff)
        - name: install-deps
          value: yarn install
        - name: run-test
          value: yarn test
      workspaces:
        - name: source
          workspace: shared-data
        - name: cache
          workspace: cache

Here we organize the logic of our pipeline.

To help with organization and reutilization of tasks, which are the more atomic resources of a Tekton pipeline, we use a cluster resolver. This way, we can have one task shared across all namespaces and eliminate the need to duplicate tasks that are common to multiple pipelines. The cluster resolver takes the namespace the task is in and the name of the task.

The parameters we define in the TriggerTemplate and pass to the pipeline run are defined in the pipeline and passed to the tasks.

Another great feature of the Tekton pipeline is the TaskResult. Notice we use a parameter in the test-task that is inherited from a task result. This result is defined in the task fetch-source, which is the task we will create to clone a remote repository. The parameter diff is a list of files that were modified in the PR that triggered this pipeline.

The workspaces we define in the TriggerTemplate are also assigned to the tasks. This ensures all pods created for all tasks in the pipeline execute our automations in the same storage space. That way, we can clone the remote repository at the beginning of the pipeline and perform many tasks with the same files.

Creating our tasks

Now we define the tasks, which are the actual work to be done in our pipeline. In Tekton, each task is a Pod in Kubernetes. It is composed of several steps, each step being a container inside this task pod.

Fetch Source Task

This task fetches a remote repository. We are using depth 2, which means we will be getting the last two commits from the repository, avoiding downloading too much data.

We will also generate some task results we can use in our pipelines. The result is defined in the Task custom resource. Tekton also provides an api to interact with results. To use the results from a task, we only need to reference them as tasks..results..

Here an example in our public repository of a fech-source task.

Test Task

This is a simple task that executes a script command you provide. As with the previous task, we define the workspace where we clone the repository and define one step to install dependencies and another to run the tests. There are many ways to organize this same scenario; this is just an example of tasks and how steps are defined.



apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
  name: task-test
  namespace: {{ .Values.projectName }}
spec:
  description: >-
    A generic task to run any bash command in any given image
  workspaces:
    - name: source
      optional: true
    - name: cache
      optional: true
  params:
    - name: run-test
      type: string
    - name: install-deps
      type: string
    - name: diff
      type: string
      description: diff of the pull request
    - name: image
      type: string
      default: "node:latest"
  steps:
   - name: install
      image: $(params.image)
      workingDir: $(workspaces.source.path)
      script: |
        #!/usr/bin/env bash
        set -xe
        $(params.install-deps)
    - name: test
      image: $(params.image)
      workingDir: $(workspaces.source.path)
      script: |
        #!/usr/bin/env bash
        set -xe
        $(params.run-test)

Dashboard

In order to visualize your tekton resources, tekton operators will create a service to host the Tekton Dashboard, to find the ip asigned to the dashboard run:



 kubectl get services -n tekton-pipelines

Final Considerations

In this article, we covered the steps to automate a simple testing pipeline using the Tekton Operator. There are many ways to achieve this same result, and Tekton is a powerful tool that offers a lot of resources you can use depending on your needs. Additionally, the community is active and supportive. You can open an issue on the Tekton Github.

Woovi is also improving our own CI/CD internal platform. In the WooviOps repo, you can find our basic implementation of the same case we covered in this article and more. If you want to help us improve, we welcome your PRs, comments, or you can just reach out to us on Twitter.

Also, Woovi is hiring!

Photo by Rock'n Roll Monkey on Unsplash

Owning your infrastructure: A Journey to Bare Metal and out of the Cloud

Caio Campos Borges Rosa — Wed, 10 Apr 2024 13:10:53 +0000

In today's evolving technological landscape, companies continually assess their infrastructure choices to adapt to changing needs. For Woovi, having the ability to rapidly test-run ideas, maintain full control of our infrastructure, and yet not overspend was the driving force behind this decision. In this article, we'll delve into Woovi's transition and explore the motivations, challenges, and outcomes of transitioning out of the cloud.

Motivations and Aspirations

Our motivations for transitioning our infrastructure stemmed from several key factors. Firstly, We aimed to reduce costs significantly while simultaneously enhancing the quality of our technological environment, which seemed almost too good to be true, but at the same time we wanted also a higher standard of infrastructure performance, faster database operations and application responsiveness, better and faster loging and debugging, we wanted it all.

AWS Cloud stack

We had a very straightforward setup on AWS; however, its performance was not particularly fast. As a result, any heavy data processing job could significantly slow us down.

EKS
It was our kubernetes service on aws, 5 t3.2xlarge nodes, performance was ok but not great. EBS for block storage as some of our data services ran on k8s.

S3
Object storage for assets.

ECR
ECR was our container registry, ECR is great in general.

Mongodb
E nodes replica set on k8s, 1 hidden replica for analytics and backups.

Redis
1 node on k8s.

Elasticsearch
1 Node running on k8s, we also ran kibana and APM integrations.

CI/CD
Github and Circleci, this part of the migration was so important and complex that we will have a separate series just on CI/CD.

Woovi Bare Metal Stack

Data center
We chose a datacenter in São Paulo with colocation services. You bring the servers they take care of it, they offer internet redundancy, generators, coolling and 24/7 support.

Network

Mikrotik routers, two of them, we wanted redundancy on the network layer. We will talk more a bout how we achieve that later. We have internal DNS services for service discovery and other network services we will soon talk more about. We also have a VRRP(Virtual Router Redundancy Protocol) setup on our Mikrotik for redundancy.

Internet

Two internet direct link providers.

Servers

Proxmox was our hypervisor solution due to its comprehensive feature set, providing us with the necessary flexibility. As a small team responsible for managing our infrastructure, Proxmox aligns perfectly with our requirements. Moreover, it's a familiar tool for our team.
Our Staging environment is hosted on a Dell T550, we run a small version of each service we have on production.
In our production environment we run 5 Dell R750.

Kubernets
We chose microk8s for its lightweight nature and comprehensive feature set. We sought simplicity, with all Kubernetes applications running in our cluster being stateless, prioritizing performance.

Mongodb
3 replicasets runnign in separate servers, on LXC containers managed by Proxmox, for disk we use 1.5TB of ssd space, CPU 15 cores and RAM 48gb. Mongodb is quite havy on memory, our focus is performance but we put a lot of effort on backup strategies.

Redis
One node running on a managed container, we have plans to expand this service but for now works great for us.

Observability

We migrated our ELK stack from 7 to 8, integrated over Elastic Agents and now we plan to expand observability using Logstash and Beats to improve Application observability.

We also added Prometheus/Grafana to our stack, we want to evolve our kubernetes observability as we scale.

For network and hardware monitoring, we have a Zabbix Server running on an LXC container.

There is a lot of overlap on those solutions, we wanted to test most of them and see what is the best suited for our needs, in the next months we will have a better understanding of those tools and this might change.

CI/CD

We created a full CI/CD platform with Tekton and ArgoCD, we will discuss in detail in its own series Cloud Native CI/CD.

Registry

We have implemented a fully managed registry using Harbor, which serves as a repository for Docker images and Helm Charts. It is seamlessly integrated into our CI/CD pipelines and Kubernetes environment. Additionally, Harbor offers the ability to integrate with Static Analysis and vulnerability tools such as Clair and Trivy.

Tools

For provisioning we use ansible, this is part of an effort to follow IaC(Infrastructure as Code) paradigm, keep a good level of visibility as we scale.

Overview

Migration

The migration was simple:

1 - Cut access to the platform using cloudflare.

2 - Scale down workers on EKS

3 - Scale down servers on EKS

4 - Backup disks

5 - Wait for replication on Mongodb Cluster

6 - Scale up Services on new infrastructure

7 - Clear access on cloudflare to new infrastructure

8 - Test absolutely everything!

!Done

The most challenging aspect of the migration was the database transition. To address this, we created proxy containers for our new replicas and incorporated them as hidden replicas within our EKS cluster. Over the course of a week-long testing period, we achieved nearly perfect replication with minimal lag. This success instilled confidence in us, assuring that downtime would be kept to a minimum, we encountered some hiccups during the migration process, particularly with outdated images on our self-hosted registry, causing a few services to be off by approximately 10 minutes. Despite these challenges, we successfully migrated the entire infrastructure in just 17 minutes.

First Week Impressions

Everything seems too fast, this was the first week we fixed bugs that only occured because our database is so much faster than before. Responsiveness on our platform, queries, observability, tooling, all we had before became much faster.

Our CI/CD is currently broken with deploys being perform manually, but we will have in no time fully integrated.

Cost was, as expected, a big win, our AWS bill range from 50k to 100k BRL before.

We will share more in depth artciles about aspects of the migration as we improove our infrastructure, if you want to help...

We are hiring!

Photo by Filipe Dos Santos Mendes on Unsplash

How Woovi is building a self hosted cloud native CI/CD platform with Tekton and Argocd

Caio Campos Borges Rosa — Tue, 09 Apr 2024 13:12:29 +0000

At Woovi, we prioritize both quality and speed in our software development processes, as we've previously discussed in our article on software quality. This commitment entails running over 100,000 tests daily and deploying multiple times to production without disruption. Achieving this level of efficiency requires substantial investment in CI/CD infrastructure. In the past, we relied on CircleCI as our preferred platform, which served us well but came with significant costs.

In this article, we'll provide an overview of our journey toward building our own CI/CD platform. This platform will not only enhance our development experience but also result in a remarkable 87% reduction in CI/CD costs within the first 30 days of implementation, compared to our previous setup with CircleCI. We're excited to share our progress and insights with you.

Requirements

The requirements for our CI/CD pipelines quite simple but not easy to achieve:

GitOps

It must follow gitOps culture, all infrastructure must be described in a git repository, that way we can have auditable changes and updates not only for our software but for infrastructrure aswell.

Automation

Changes in software must trigger CI/CD pipeline automatically, without human intervention.

Conditional Testing

Efficient resource allocation is crucial. Therefore, only tests relevant to the modified portion of the software should be executed. This minimizes unnecessary resource consumption and time spent.

Full Suite Testing

Must be able to run full test suites, our monorepos have 7-10k tests each. We use jest as our test framework, that means we have to optimize for it.

All self hosted

No cloud services, no paid platforms.

Cloud Native

Taking advantage of the k8s api and operators, we can use the controll loop to manage our CI/CD applications. Tekton is a very powerfull cloud native tool we will be using for this project.

Design

The CI/CD flow components are:

Events

The events are generated from github, each push of new code a developer performs have to trigger a test pipeline. Deploy requests are created from a pull request with the release tags.

Pipelines

One pipeline for each repository, the pipeline will run tests, build applications, update registry and bucket with the new version of software if the tests pass and deploy in the correct enviroment based on the event who triggered the pipeline.

Applications

Pipelines will interact with applications directly, deploying new versions, performing rollbacks if something goes wrong and sending notifications about overall status.

Tekton and ArgoCD

Tekton is a very powerfull cloud native CI/CD framework, is open source and has very strong comunity support. We choose mainly because it has all we need in terms of features, with some still in development, and the comunity is very active and helpfull.

Argocd is a declarative GitOps tool, it allow us to implement track our helm charts from repository changes. We also implement the deploy stage of our production enviroment using argocd apis, that from a task in tekton can sync applications in kubernetes after we update our registry. Its also very complete with many security features out of the box.

We have a dedicated server for our staging enviroment that runs proxmox hypervisor, there we run our kubernetes cluster in a VM. We will do a deep dive on upcoming articles for each building block and hope to show a little of how we setup the platform.

Initial Impact

We saw a whopping reduction of R$40,000.00 in monthly CI/CD costs!

The primary expense in our CI/CD setup lies in the development pipelines. In our initial implementation, we opted to retain the production pipelines while migrating only the development ones. The outcome? An impressive 87% reduction in costs.

Not all impact was positive, our pipelines are taking 2x the time run as off the last implementation, we have a lot to develop in order to match circleci features. Performance too took a big hit, turns out running so many tests using jest cost alot of resources. Our staging server when running full test pipelines is always on 95% capacity.

Next article we will be covering, the steps involved in setting up Tekton and Argo CD within Woovi's Kubernetes cluster or any k8s cluster.

In the meantime, we've just published our tools and manifests on GitHub WooviOps. Feel free to contribute, and let us know if we can improve anything.

Woovi
Woovi is a Startup that enables shoppers to pay as they like. To make this possible, Woovi provides instant payment solutions for merchants to accept orders.

If you want to work with us, we are hiring!

Photo by Erlend Ekseth on Unsplash

How to setup local development with ELK observability tools using docker-compose

Caio Campos Borges Rosa — Mon, 05 Feb 2024 12:26:17 +0000

A lot of observability is done using cloud provider tools that are one subscription away, yet replicating them for local development can be challenging. At Woovi, we prioritize aligning our development environment with production. That's why we run locally all services available in production using Docker Compose. Observability is not an exception. In this article, we'll delve into the detailed setup of ELK observability tools in our local development environment.

ELK Tools

This are the tools we use for observability in our stack.

APM
Application performance monitoring.
Elastic Search
Search Engine.
Kibana
Visualization Platform
Logstash
Log ingestion
MetricBeat
Server data shipper
Filebeat
Log harvester and aggregator
Fleet
Agents integration

Elastic Integrations offers an unified way to ingrate features of ELK stack, managed by fleet servers you can integrate over Kibana all integrations available using elastic agents, and manage them using policies trough an unified interface. In order to use this features we need security enable in kibana and elastic search and we need to provide certificate authentication for each service. In order to simplify we will be using the same certificate for all services, but this is only suited for local development and should not be used in production.

First we will need configuration files for some services, the file structure will be:

├── .env

├── docker-compose.yml

├──|observability |──  filebeat.yml

├── |observability |── logstash.conf

└── |observability |── metricbeat.yml

Docker Compose

The docker compose will be responsible for setting up certificates and copying to the volume of each service, that way we can setup SSL between services. You can find the .env file on the github repo

version: "3.8"
services:
  setup:
    image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
    volumes:
      - certs:/usr/share/elasticsearch/config/certs
    user: "0"
    command: >
      bash -c '
        if [ x${ELASTIC_PASSWORD} == x ]; then
          echo "Set the ELASTIC_PASSWORD environment variable in the .env file";
          exit 1;
        elif [ x${KIBANA_PASSWORD} == x ]; then
          echo "Set the KIBANA_PASSWORD environment variable in the .env file";
          exit 1;
        fi;
        if [ ! -f config/certs/ca.zip ]; then
          echo "Creating CA";
          bin/elasticsearch-certutil ca --silent --pem -out config/certs/ca.zip;
          unzip config/certs/ca.zip -d config/certs;
        fi;
        if [ ! -f config/certs/certs.zip ]; then
          echo "Creating certs";
          echo -ne \
          "instances:\n"\
          "  - name: es01\n"\
          "    dns:\n"\
          "      - es01\n"\
          "      - localhost\n"\
          "    ip:\n"\
          "      - 127.0.0.1\n"\
          "  - name: kibana\n"\
          "    dns:\n"\
          "      - kibana\n"\
          "      - localhost\n"\
          "    ip:\n"\
          "      - 127.0.0.1\n"\
          "  - name: fleet-server\n"\
          "    dns:\n"\
          "      - fleet-server\n"\
          "      - localhost\n"\
          "    ip:\n"\
          "      - 127.0.0.1\n"\
          > config/certs/instances.yml;
          bin/elasticsearch-certutil cert --silent --pem -out config/certs/certs.zip --in config/certs/instances.yml --ca-cert config/certs/ca/ca.crt --ca-key config/certs/ca/ca.key;
          unzip config/certs/certs.zip -d config/certs;
        fi;
        echo "Setting file permissions"
        chown -R root:root config/certs;
        find . -type d -exec chmod 750 \{\} \;;
        find . -type f -exec chmod 640 \{\} \;;
        echo "Waiting for Elasticsearch availability";
        until curl -s --cacert config/certs/ca/ca.crt https://es01:9200 | grep -q "missing authentication credentials"; do sleep 30; done;
        echo "Setting kibana_system password";
        until curl -s -X POST --cacert config/certs/ca/ca.crt -u "elastic:${ELASTIC_PASSWORD}" -H "Content-Type: application/json" https://es01:9200/_security/user/kibana_system/_password -d "{\"password\":\"${KIBANA_PASSWORD}\"}" | grep -q "^{}"; do sleep 10; done;
        echo "All done!";
      '
    healthcheck:
      test: ["CMD-SHELL", "[ -f config/certs/es01/es01.crt ]"]
      interval: 1s
      timeout: 5s
      retries: 120

  es01:
    depends_on:
      setup:
        condition: service_healthy
    image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
    labels:
      co.elastic.logs/module: elasticsearch
    volumes:
      - certs:/usr/share/elasticsearch/config/certs
      - esdata01:/usr/share/elasticsearch/data
    ports:
      - ${ES_PORT}:9200
    environment:
      - node.name=es01
      - cluster.name=${CLUSTER_NAME}
      - discovery.type=single-node
      - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
      - bootstrap.memory_lock=true
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.key=certs/es01/es01.key
      - xpack.security.http.ssl.certificate=certs/es01/es01.crt
      - xpack.security.http.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.key=certs/es01/es01.key
      - xpack.security.transport.ssl.certificate=certs/es01/es01.crt
      - xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.license.self_generated.type=${LICENSE}
    mem_limit: ${ES_MEM_LIMIT}
    ulimits:
      memlock:
        soft: -1
        hard: -1
    healthcheck:
      test:
        [
          "CMD-SHELL",
          "curl -s --cacert config/certs/ca/ca.crt https://localhost:9200 | grep -q 'missing authentication credentials'",
        ]
      interval: 10s
      timeout: 10s
      retries: 120

  kibana:
    depends_on:
      es01:
        condition: service_healthy
    image: docker.elastic.co/kibana/kibana:${STACK_VERSION}
    labels:
      co.elastic.logs/module: kibana
    volumes:
      - certs:/usr/share/kibana/config/certs
      - kibanadata:/usr/share/kibana/data
      - ./observability/kibana.yml:/usr/share/kibana/config/kibana.yml:ro
    ports:
      - ${KIBANA_PORT}:5601
    environment:
      - SERVERNAME=kibana
      - ELASTICSEARCH_HOSTS=https://es01:9200
      - ELASTICSEARCH_USERNAME=kibana_system
      - ELASTICSEARCH_PASSWORD=${KIBANA_PASSWORD}
      - ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES=config/certs/ca/ca.crt
      - XPACK_SECURITY_ENCRYPTIONKEY=${ENCRYPTION_KEY}
      - XPACK_ENCRYPTEDSAVEDOBJECTS_ENCRYPTIONKEY=${ENCRYPTION_KEY}
      - XPACK_REPORTING_ENCRYPTIONKEY=${ENCRYPTION_KEY}
      - XPACK_REPORTING_KIBANASERVER_HOSTNAME=localhost
      - SERVER_SSL_ENABLED=true
      - SERVER_SSL_CERTIFICATE=config/certs/kibana/kibana.crt
      - SERVER_SSL_KEY=config/certs/kibana/kibana.key
      - SERVER_SSL_CERTIFICATEAUTHORITIES=config/certs/ca/ca.crt
      - ELASTIC_APM_SECRET_TOKEN=${ELASTIC_APM_SECRET_TOKEN}
    mem_limit: ${KB_MEM_LIMIT}
    healthcheck:
      test:
        [
          "CMD-SHELL",
          "curl -I -s --cacert config/certs/ca/ca.crt https://localhost:5601 | grep -q 'HTTP/1.1 302 Found'",
        ]
      interval: 10s
      timeout: 10s
      retries: 120

  metricbeat01:
    depends_on:
      es01:
        condition: service_healthy
      kibana:
        condition: service_healthy
    image: docker.elastic.co/beats/metricbeat:${STACK_VERSION}
    user: root
    volumes:
      - certs:/usr/share/metricbeat/certs
      - metricbeatdata01:/usr/share/metricbeat/data
      - "./observability/metricbeat.yml:/usr/share/metricbeat/metricbeat.yml:ro"
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
      - "/sys/fs/cgroup:/hostfs/sys/fs/cgroup:ro"
      - "/proc:/hostfs/proc:ro"
      - "/:/hostfs:ro"
    environment:
      - ELASTIC_USER=elastic
      - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
      - ELASTIC_HOSTS=https://es01:9200
      - KIBANA_HOSTS=https://kibana:5601
      - LOGSTASH_HOSTS=http://logstash01:9600
      - CA_CERT=certs/ca/ca.crt
      - ES_CERT=certs/es01/es01.crt
      - ES_KEY=certs/es01/es01.key
      - KB_CERT=certs/kibana/kibana.crt
      - KB_KEY=certs/kibana/kibana.key
    command: -strict.perms=false

  filebeat01:
    depends_on:
      es01:
        condition: service_healthy
    image: docker.elastic.co/beats/filebeat:${STACK_VERSION}
    user: root
    volumes:
      - certs:/usr/share/filebeat/certs
      - filebeatdata01:/usr/share/filebeat/data
      - "./filebeat_ingest_data/:/usr/share/filebeat/ingest_data/"
      - "./observability/filebeat.yml:/usr/share/filebeat/filebeat.yml:ro"
      - "/var/lib/docker/containers:/var/lib/docker/containers:ro"
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
    environment:
      - ELASTIC_USER=elastic
      - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
      - ELASTIC_HOSTS=https://es01:9200
      - KIBANA_HOSTS=https://kibana:5601
      - LOGSTASH_HOSTS=http://logstash01:9600
      - CA_CERT=certs/ca/ca.crt
    command: -strict.perms=false

  logstash01:
    depends_on:
      es01:
        condition: service_healthy
      kibana:
        condition: service_healthy
    image: docker.elastic.co/logstash/logstash:${STACK_VERSION}
    labels:
      co.elastic.logs/module: logstash
    user: root
    volumes:
      - certs:/usr/share/logstash/certs
      - logstashdata01:/usr/share/logstash/data
      - "./logstash_ingest_data/:/usr/share/logstash/ingest_data/"
      - "./observability/logstash.conf:/usr/share/logstash/pipeline/logstash.conf:ro"
    environment:
      - xpack.monitoring.enabled=false
      - ELASTIC_USER=elastic
      - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
      - ELASTIC_HOSTS=https://es01:9200

  fleet-server:
    depends_on:
      kibana:
        condition: service_healthy
      es01:
        condition: service_healthy
    image: docker.elastic.co/beats/elastic-agent:${STACK_VERSION}
    volumes:
      - certs:/certs
      - fleetserverdata:/usr/share/elastic-agent
      - "/var/lib/docker/containers:/var/lib/docker/containers:ro"
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
      - "/sys/fs/cgroup:/hostfs/sys/fs/cgroup:ro"
      - "/proc:/hostfs/proc:ro"
      - "/:/hostfs:ro"
    ports:
      - ${FLEET_PORT}:8220
      - ${APMSERVER_PORT}:8200
    user: root
    environment:
      - SSL_CERTIFICATE_AUTHORITIES=/certs/ca/ca.crt
      - CERTIFICATE_AUTHORITIES=/certs/ca/ca.crt
      - FLEET_CA=/certs/ca/ca.crt
      - FLEET_ENROLL=1
      - FLEET_INSECURE=true
      - FLEET_SERVER_ELASTICSEARCH_CA=/certs/ca/ca.crt
      - FLEET_SERVER_ELASTICSEARCH_HOST=https://es01:9200
      - FLEET_SERVER_ELASTICSEARCH_INSECURE=true
      - FLEET_SERVER_ENABLE=1
      - FLEET_SERVER_CERT=/certs/fleet-server/fleet-server.crt
      - FLEET_SERVER_CERT_KEY=/certs/fleet-server/fleet-server.key
      - FLEET_SERVER_INSECURE_HTTP=true
      - FLEET_SERVER_POLICY_ID=fleet-server-policy
      - FLEET_URL=https://fleet-server:8220
      - KIBANA_FLEET_CA=/certs/ca/ca.crt
      - KIBANA_FLEET_SETUP=1
      - KIBANA_FLEET_USERNAME=elastic
      - KIBANA_FLEET_PASSWORD=${ELASTIC_PASSWORD}
      - KIBANA_HOST=https://kibana:5601

volumes:
  elastic_data: {}
  certs:
    driver: local
  esdata01:
    driver: local
  kibanadata:
    driver: local
  metricbeatdata01:
    driver: local
  filebeatdata01:
    driver: local
  logstashdata01:
    driver: local
  fleetserverdata:
    driver: local
  cache:
    driver: local

networks:
  default:
    name: elastic
    external: false

Its not as complicated as it looks, some of the things the compose file does by service are:

Setup

Create an instance of an Elastic Search service to use on validation of the certificates. Then, via bash script we create the certificates using elasticsearch-certutil, here we specify what are our cluster nodes and servers instances creating CA cert and node certs giving the correct file permissions and distributing the cert and keys. We test the authentication worked using elastic search service and then the service stops.

ES01

Main Elastic Search service node, xpack.security options need to be enabled and pointing to the correct folder with the node certificate and certificate authority. A basic health check is passed using auth validation as main test parameter.

Kibana

Kibana service, security config is the same as ES, xpack need to be enabled, extra configuration will be passed in its own yml file. To Access this file we pass it as bind read only volume.
./observability/kibana.yml:/usr/share/kibana/config/kibana.yml:ro
We add internal apm traces so once the initial setup go up we can verify kibana for itself.

filebeat01

For filebeat we bind the volumes /var/lib/docker/containers and /var/run/docker.sock allowing filebeat to get all containers internal logs, the config in observability/filebeat.yml will set autodiscover for that purpose, using docker as a provider, will also config kibana and ES. Ingest data input and ES outupt.

metricbeat01

For Metricbeat the binding mounts are to allow getting host system metrics and information and sending to elastic search.

logstash01

For logstash the bind mount config file will setup ingest plugins and output to elasti search. We are using basic starter configuration but in the config file is where you will setup formating. The ingest data path you can send the log file to be read using the strategy described in the config file.

fleet-server

Fleet server will manage agents and its integrations.

Running Stack

1 - Run:

docker compose up -d

Wait untill is finished:

2 - Open kibana

Enter username and password setup in .env file

3 - Go to Fleet/Management

4 - Go to Settings

5 - Edit the output agent

You will need 3 items to configure the certificate.

Url: https://es01:9200

Elastic CA trusted Fingerprint:

To get this value you need to sign the certificate from ES container

Firtst copy the certificate to a temp folder

docker cp woovi-es01-1:/usr/share/elasticsearch/config/certs/ca/ca.crt /tmp/.

Then sign the certificate getting the Fingerprint

openssl x509 -fingerprint -sha256 -noout -in /tmp/ca.crt | awk -F"=" {' print $2 '} | sed s/://g

The output will be something like this:

5A7464CEABC54FA60CAD3BDF16395E69243B827898F5CCC93E5A38B8F78D5E7

Use the value on the field Elastic CA trusted Fingerprint

Now in Advanced YAML configuration you will need the certificate.

Print the certificate you copied from the container

cat /tmp/ca.crt

And edit to match the format on this example file
Its important not to input the wrong indentation, yml really sucks and this step is usually to blame when something goes wrong.

Save and apply settings, wait a few seconds and you should be able to see system metrics like memory and cpu usage on the agent page here

That indicates your fleet server is configured the correct way.

Now in your local development you only need to use apm config, go over on the agent configuration for APM in your desired language and check out, most are like the following:

{
  serviceName: 'my-service-name',
  secretToken: 'supersecrettoken',
  serverUrl: 'http://localhost:8200',
  environment: 'my-environment'
}

Check out for aditional information in this agent integration page

We're constantly seeking ways to automate and streamline processes. If you have any ideas on how to enhance this workflow, please don't hesitate to share them with us!

Woovi
Woovi is a Startup that enables shoppers to pay as they like. To make this possible, Woovi provides instant payment solutions for merchants to accept orders.

If you want to work with us, we are hiring!

Automate homelab microK8s cluster provisioning with Vagrant and Ansible

Caio Campos Borges Rosa — Mon, 05 Feb 2024 11:47:54 +0000

Getting quick feedback is essential while developing. When it comes to setting up infrastructure, it often involves tons of scripting and is good to have a simple workflow you can iterate on. This tutorial is all about giving you a solid head start if you're itching to dive into DevOps and build your own homelab. We'll be installing microk8s, a fully compliant and up-to-date Kubernetes distribution that boasts minimal machine requirements.

To manage our VMs we will use Vagrant, its a complete cli tool to manage virtual machines and provides a simple but complete workflow. Vagrant is very good at giving us a context to our project, instead of configuring each VM individually and running provisioning scripts against them we have one declarative file that describe the project.

Vagrant installation

$ brew install virtualbox

$ brew cask install vagrant

Vagrantfile

Vagrant uses a Vagrantfile to describe the machines the project is going to need. Here we will have our provisioning script that will install update repositories, install Ansible, and setup host records. For each machine we describe we will run the provisioning script when we call "vagrant up ".

# -*- mode: ruby -*-
# vi: set ft=ruby :
$script = <<-SCRIPT
apt-get update
apt-get install -y ansible sshpass
echo "192.168.56.11 controller" >> /etc/hosts
echo "192.168.56.12 node-1" >> /etc/hosts
SCRIPT
Vagrant.configure("2") do |config|
  config.vm.define "controller" do |controller|
  controller.vm.box = "ubuntu/focal64"
  controller.vm.network "private_network", ip: "192.168.56.11"
  controller.vm.hostname = "controller"
  controller.vm.provider "virtualbox" do |vb|
    vb.memory = "2048"
  end
 controller.vm.provision "shell", inline: $script
end
config.vm.define "node1" do |node1|
  node1.vm.box = "ubuntu/focal64"
  node1.vm.network "private_network", ip: "192.168.56.12"
  node1.vm.hostname = "node1"
  node1.vm.provider "virtualbox" do |vb|
   vb.memory = "1024"
  end
  node1.vm.provision "shell", inline: $script
end
end

NOTE: if you need a ip range out of vagrant default '192.168.56.0/21', create a file '/etc/vbox/networks.conf' and add the line with the range you need. Ex: * 192.168.32.0/24.

Next we call vagrant up:

$ vagrant up controller

If you already run and need to make changes to any VM described in the Vagrantfile, run the command again with the flag --provision, that will run the provisioning script again for the targeted VM.

Ansible playbook

For k8s provisioning we will be using a Ansible. Ansible is a powerful automation to provision and manage resources, we can automate rolling updates and state management of our deployments in a declarative file. Here we will use in its simplest form to install microk8s and add alias to kubectl and completion using a playbook.

---
- name: Install Microk8s
  hosts: localhost
  gather_facts: false
  become: true
  tasks:
    - name: Install microk8s
      snap:
        name: microk8s
        state: present
        classic: yes
    - name: Add alias to kubectl
      become: false
      lineinfile:
        path: '{{ lookup("env", "HOME") }}/.bashrc'
        regexp: '^alias kubectl='
        line: 'alias kubectl="microk8s kubectl"'
        state: present
    - name: Add bash completion for kubectl
      become: false
      lineinfile:
        path: '{{ lookup("env", "HOME") }}/.bashrc'
        regexp: '^source \<\(kubectl'
        line: 'source <(kubectl completion bash)'
        state: present

Now we just need to access the vm using ssh, Vagrant already setup a user with ssh access to the machines, so we only need to run:

vagrant ssh controller

Ansible is already installed, we did that in the provisioning script so now we only need to run the playbook:

$ ansible-playbook /vagrant/homelab-microk8s.yml

After ansible finish to run the playbook, we need to source the .bashrc, you can just log out wait 2s and log back in with vagrant ssh:

$ exit

$ vagrant ssh controller

Now you can use kubectl with completion and get info of the services running:

$ kubectl get nodes
NAME         STATUS   ROLES    AGE   VERSION
controller   Ready    <none>   60m   v1.28.3

$ kubectl get pods -A

NAMESPACE     NAME                                     READY   STATUS    RESTARTS   AGE
kube-system   calico-node-sfngb                        1/1     Running   0          61m
kube-system   coredns-864597b5fd-drm79                 1/1     Running   0          60m
kube-system   calico-kube-controllers-77bd7c5b-vnx5j   1/1     Running   0          60m

references:
https://developer.hashicorp.com/vagrant/docs/vagrantfile
https://docs.ansible.com/ansible/latest/playbook_guide/playbooks_intro.html

Here you will find a repo with the files used in this guide.

Photo by Jordan Harrison on Unsplash

How to effectively use retry policies with BullJs/BullMQ

Caio Campos Borges Rosa — Wed, 18 Oct 2023 01:44:51 +0000

Jobs play a pivotal role in the majority of distributed systems these days. They allow us to achieve scalability at the cost of time. While it may take more time to process all the tasks in our queue, rest assured that we will process each one eventually. Except, if they fail.

In distributed systems, it's generally a good practice to treat events as disposable and immutable. Unless you have a specific use case that necessitates storing events, this should be your default approach.

How to deal with failed jobs

When a job fails, there are several ways to address the situation. Reprocessing is often the first option that comes to mind, but it typically involves recreating the event, requiring user input, or asking the user to repeat an action. This can be detrimental to the system and user experience since unnecessary repetition is generally undesirable.

What is a retry policy?

Another approach to handling failing jobs that require reprocessing is through the use of retry policies. A retry policy comprises a set of rules that automate job reprocessing. Typically, these rules define the time intervals between each retry and the maximum number of retry attempts.

How to configure in BullMQ/JS?

In BullMQ/BullJS, you configure this directly when adding a job to a queue using the job options object during job creation.

await queue.add(
  'test-retry',
  { foo: 'bar' },
  {
    attempts: 3,
    backoff: {
      type: 'exponential',
      delay: 1000,
    },
  },
);

The 'attempts' parameter determines how many times a job will be retried, starting from the first failure. The 'backoff' function involves a dynamic relationship between 'attempts' and the time intervals, which is described using 'type' and 'delay'."

The 'type' specifies the strategy for handling the retry attempts, while 'delay' represents the time intervals between these attempts.

Fixed

When the backoff type is set to 'fixed,' the retry attempts will be evenly spaced after the specified delay time. For instance, as shown in the example above, Bull will make 7 attempts, waiting 10 seconds between each one.

 {
        attempts: 7,
        backoff: {
          type: 'fixed',
          delay: 10000,
        },
      },

Exponential

When the backoff type is set to 'exponential,' the retry attempts will follow an exponential pattern after the specified delay time. For example, as demonstrated in the above example, Bull will make 7 attempts, waiting 10 seconds after the first, 20 seconds after the second, 30 seconds after the third, and so on.

 {
        attempts: 7,
        backoff: {
          type: 'exponential',
          delay: 10000,
        },
      },

If the only issue you are trying to solve is handling errors and retries automatically that would be enough, but what if you want to control when a job fails?

How to explicit fail a job?

The retry policy will only apply to jobs that explicitly fail. Jobs are considered failed when they've reached their maximum number of stalls (refer to the documentation) or when an error occurs while a worker is processing the job.

You can create custom error classes and explicitly instantiate them based on logic your code controls.

For example, if I make a request and the status code is anything other than 200:

My error class:

export class WorkerRetryError extends Error {
  constructor(message) {
    super(message);
    this.name = 'WorkerRetryError';
  }
}

export const isWorkerRetryError = (err: Error) =>
  err instanceof WorkerRetryError;

Using error class in this case is important, so you can use narrowing with typeguards at runtime to clear the custom errors you will be trowing.

 const getHasError = () => {
    if (result?.error) {
      return true;
    }
    if (result?.response?.status > 200) {
      return false;
    }

    return !result?.response?.ok;
  };

  const hasError = getHasError();

  if (hasError) {
    throw new WorkerRetryError(
      `Webhook failed with status code 
        ${result?.response?.status}`,
    );
  }

Photo by Mark König on Unsplash

The right decisions can simplify and make it easier to scale

Caio Campos Borges Rosa — Mon, 07 Aug 2023 01:07:53 +0000

A couple of weeks ago, I talked about how Woovi uses MongoDB Change Streams to emit events in our event-driven architecture. Now, I'm going to talk about the repercussions of this kind of architectural decision and how much value lies solely in decision-making and good software development principles.

Principle

Principles help narrow down options when deciding how to implement a new feature or adopt a new technology for a feature..
At Woovi, we adopt an event-driven architecture to scale our platform. This means that in every design decision, we consider whether it aligns with this perspective.

Is there an organic event for that implementation, meaning the user produces some behavior that can be captured as an event? Or do we need to produce an event based on what we want to happen?
It has to process fast or is costly?
It needs to happen as fast as possible or can be delayed?
It needs to be atomic?
It needs to be idempotent?

We can take a different approach for each of those questions, but it is common for the process to be very similar for every implementation due to the event-driven principle.

The problem and decision

Our platform is very data heavy and we always wanted to offer the best search feature for our users, being able to find anything from a single search bar is our goal with this feature, for that we decided on using elastic search to power our search solution.
The first problem we encountered was, how to index all our data?

First we need to be event driven, so no cronJob pooling all of our data from db and indexing.
Second we need to be fast, as fast as data is created in our platform it needs to be searchable for the user. We deal with financial transactions, speed is essencial. This two points complement each other, if i need to be event driven i have to find a way to index data as data is created on our platform.
Third, we can't create a point of stress on our system, so it needs to be lean. We can't tax our database with heavy queries in order to create events.
And last we need to be resilient, we need to deliver all the data the user wants access to.

The solution was to create events from the database itself.

The publisher

A couple of weeks ago, we talked about this package, and it has matured a lot since then. Now, it consists of a changeStreamListen method that takes a mongoose Model for the change stream and a function that will receive the data from the stream. This approach allows us to scale by adding multiple methods inside this function for the same model and different types of streams.

import type {
  ChangeStreamDeleteDocument,
  ChangeStreamUpdateDocument,
  ChangeStreamInsertDocument,
} from 'mongodb';
import type { Model } from 'mongoose';

import { changeStreamMiddleware } from './changeStreamMiddleware';

export type AllowedChangeStreams =
  | ChangeStreamInsertDocument
  | ChangeStreamUpdateDocument
  | ChangeStreamDeleteDocument;

export type OperationType = AllowedChangeStreams['operationType'];

export type ChangeStreamData<T> = {
  _id: Record<string, unknown>;
  operationType: OperationType;
  wallTime: Date;
  fullDocument: T;
  ns: Record<string, unknown>;
};

export const changeStreamListen = <T>(model: Model<T>, fn: (data: ChangeStreamData<T>) => void) => {
  const stream = model.watch([], { fullDocument: 'updateLookup' });

  stream.on('change', changeStreamMiddleware(model.collection.name, fn));
}

Notice the changeStreamMiddleware method encapsulating the model and the function? This is a Higher-Order Function (HoF) designed to attach APM (Application Performance Monitoring) and index transactions in our observability tools, reducing the risk of intrusion or failure for the processes being watched. More on that in a future article.

After that, we have setupSubscribers, which instantiates one subscriber for each model we want to watch and passes a function to handle the data from the events.

import { Charge } from '@woovi/charge';
import { Company } from '@woovi/company';
import { Customer } from '@woovi/customer';
import { PixTransaction } from '@woovi/transaction';
import { User } from '@woovi/user';

import { changeStreamListen } from './changeStreamListen';
import { chargeChangeStreamHandler } from '../charge/chargeChangeStreamHandler';
import { handleCompanySubscriberEvent } from '../company/handleCompanySubscriberEvent';
import { handleCustomerSubscriberEvent } from '../customer/handleCustomerSubscriberEvent';
import { handleTransactionSubscriberEvent } from '../transaction/handleTransactionSubscriberEvent';
import { handleUserSubscriberEvent } from '../user/handleUserSubscriberEvent';

export const setupSubscribers = () => {
  changeStreamListen(Charge, chargeChangeStreamHandler);
  changeStreamListen(PixTransaction, handleTransactionSubscriberEvent);
  changeStreamListen(Customer, handleCustomerSubscriberEvent);
  changeStreamListen(User, handleUserSubscriberEvent);
  changeStreamListen(Company, handleCompanySubscriberEvent);
};

This implementation performs really well and delivers events very fast as soon as it hits consensus between replica members.

This allows us to achieve one of the goals for the search implementation: fast indexing times. It takes less than one second for a transaction or charge to be available in Elasticsearch from a stream event.

It is very resilient when compared with other event-driven implementations.

We use BullJs as our event bus and worker implementation. One concept we always consider while working with jobs is that a job can always fail. For that reason, each queue has its own retry and rate policies. The publisher is intended to be used for processes that require both speed and resilience in execution.

While using jobs, you always have to implement two sides of the processes: the creation of the event and the consuming part. By using the publisher, we can simplify the complexity of our processes and become even more resilient against unintended effects.

Benefits and Concerns of using Change Data Capture

Almost every database has some kind of event capture implementation. There are many ways in which you can achieve CDC (Change Data Capture), but log-based approaches are generally considered the better choice. Examples of log-based approaches include write-ahead logs in PostgreSQL, MySQL binary logs, or MongoDB oplog. These methods offer more benefits than simply using database triggers or even worse a query based approach. They are usually more reliable since they utilize the database driver to perform the necessary security checks before emitting an event. Additionally, log-based approaches have a low impact on database performance, especially in the case of MongoDB where it is close to zero. Moreover, these methods typically have a low cost of implementation and add minimal complexity.

On the other hand, the downsides of using this approach include the lack of information on how the change takes place and its context. Much of the logic is obscured by the database layer, making it inaccessible from the publisher's side and limiting the implementation from that perspective. Other important point is change streams are infinite; as long as you are watching for changes in the model, they will be continuously streamed. Hence, it becomes crucial to adhere to software development principles and good practices more seriously.

As every single event on your database is being streamed, every query you do as a result of that is even more important and has to be optimized with that in mind. It's quite trivial to create a loop on an update event, but those are easy to avoid. On the other hand, heavy and unnecessary queries are easy to overlook. So keep that in mind: simple and lean code is best for this kind of implementation.

Enabling more Products

While jobs are a great way to scale asynchronous processes, it is possible to have too many, very fast. This is a common problem in event-driven systems. Woovi has more than 100 jobs, and we manage well with a very mature codebase. However, the problem is that it does not scale forever. To move forward, we need more ways to scale events so we can implement more products for our users.
This implementation has enabled not just our search project (which is soon moving out of beta) but much more to come.

Now, we have a strong base to implement:

Mqtt integration
We are implementing Mqtt to integrate our Maquininha and solve an old problem with Windows machines. This will allow our customers to print QR codes with ease using any commercial thermal printer.
PubSub
We are implementing real-time updates to charges and transactions in our platform. This way, customers can be notified live and track information in real time.
Analytics
We are creating a complete analytics pipeline to offer data-driven insights to ourselves and our customers. This implementation uses the publisher and jobs to push data to multiple pipelines.

And much more to come.

We are shipping a lot every day. If you are interested in joining a fast-growing startup, we are hiring!

Photo by Possessed Photography on Unsplash

Stack shopping and repo setup

Caio Campos Borges Rosa — Wed, 02 Aug 2023 22:02:41 +0000

In the last article, we discussed the vision for the product and outlined its basic functionalities to guide development. In this article, I will delve into the first steps and fundamental tasks required, as well as showcase our basic development environment.

Chores

A couple of things to consider when building a company are tax-related matters, legislation, IP registration, and domain purchase. I have already taken care of most of these aspects, but I won't focus on that here since they can vary significantly depending on your location. The truth is, you need a lot less than you might imagine to start, so the key is to focus on getting started and figure out the more tedious parts later.

Stack Shopping

You are a new company, you have no friends, you need to move fast and alone(not me because i have you guys), it's essential to move quickly and independently. For the MVP, prioritize the stack that allows you to move faster, as long as it meets the minimum functionality requirements and the available tools are sufficient. Speed should always be the determining factor to break any ties in your decision-making process.

The main part of our product will consist of one simple API and one simple web app. Given the constraint of having limited staff, our priority is to keep the development process concise and facilitate Developer Experience (DX), testing, and iteration. For these reasons, I have decided to use TypeScript for both the API and the app since it is the stack I work with daily and allows for seamless context switching without compromising development experience. Additionally, we can utilize a monorepo to keep our platform organized, and this decision becomes even more evident when considering all the reasons I previously mentioned.

Code Cohesion
Simplified DX
Simplified Testing Environment
Type sharing
Resource Sharing

The primary objective is simplicity, so we've decided to use KoaJS as our API framework for the back-end. KoaJS is a lightweight and straightforward choice that simply works. This API will handle fundamental user and business logic. If we encounter scenarios that require more specialized functionalities, we can create new packages within the monorepo and import them here.


import Koa, { Request } from 'koa';
import Router from 'koa-router';
import bodyParser from 'koa-bodyparser';
import cors from '@koa/cors';


const app = new Koa();
const router = new Router();
1
  app.use(cors({ credentials: true }));
  app.use(bodyParser());
  app.use(router.routes());
  app.use(router.allowedMethods());

  //// healthcheck endpoint
  router.get('/', async (ctx: { path: string; body: string; }, next: () => any) => {
    if (ctx.path === '/') {
      ctx.body = 'lets map!';
      return;
    }
    await next();
  });

export default app;

Regarding the web app, I've selected Next.js because of its simplicity and ease of deployment. It's straightforward and provides a smooth development experience.

DX

Currently, our development experience is kept simple. To run the project seamlessly in our local development environment, we will utilize TurboRepo and Docker. With just one command, we can have the project up and running smoothly.

To run with Turborepo just run:


pnpm run dev

To run the project with docker, have docker and docker-composed installed and run:


docker-compose up -d

Docker setup in a monorepo

In a monorepo, there are several ways to set up deployment depending on your specific requirements. For now, we'll opt for a simple approach: using one Docker container for each package. We will create a docker-compose file to coordinate these packages and ensure they communicate through a Docker network.

This setup allows us to manage and deploy each package independently while maintaining clear communication between them through the designated Docker network. As we progress, we can explore more complex deployment strategies, but for the initial stages, this straightforward method should suffice.


version: "3"

services:
  web:
    container_name: web
    build:
      context: .
      dockerfile: ./apps/web/Dockerfile
    restart: always
    ports:
      - 3000:3000
    networks:
      - app_network

  api:
    container_name: api
    build:
      context: .
      dockerfile: ./apps/api/Dockerfile
    restart: always
    ports:
      - 3001:3001
    networks:
      - app_network


networks:
  app_network:

As we move forward, we'll have the opportunity to implement some exciting building blocks that will enhance our development process and productivity. With the right tools and approaches, we can accelerate our progress, making the development experience more enjoyable and efficient for everyone involved.

Next steps are:

Setting up DataBase
Basic Api Endpoints
Testing Environment
Basic CI

The repository for this project is at github

The repository is public, so you can check all the issues and pull requests to track the project's progress. Feel free to suggest or even make a contribution.

Find me on twitter for a good time and lets build something.

Photo by Mike Petrucci on Unsplash

Building a business out of Maps and Statistics. What 'WILL' go wrong?

Caio Campos Borges Rosa — Mon, 17 Jul 2023 11:59:03 +0000

Since I remember understanding the concept of software, built by someone to somebody who need something i am deeply fascinated by it.

I am not new to the whole game of building business, being part of both failures and successes im somehow aware of what a business need to achieve minimal success. What i am new is the concept of building in public, learning in public and from the start using the public attention as both, leverage to grow a organic audience to what can be a successful business and creating an early case for skin in the game as public failures are quite hurtful have significant moral cost.

So with this series of articles i want to achieve the following goals.

Build a simple viable business(make some money maybe?)
Learn as much as i can
Share experiences so others can learn as well
Make some money?

The idea and motivation

Most ideas I have for projects are, in fact, my wife's ideas. She is a Professor of Environmental Psychology at the Federal University of Roraima (UFRR), and she loves to tell me what I should build with software.

This time, she needs to gather a huge list of statistical data from schools in major cities of Brazil to study the impact of green areas in school life.

I believe good products are those that solve real problems we have in our lives so my motivation aligns with hers for this one, and I really think that would be a great product to use not just in academic life but in many other areas.

Product

The product I am going to develop is a simple webpage that delivers statistical data based on an area the user draws in a map widget. That's it, map to statistics will be the basis for this product.

The website will consist of two sections after users login. First, they will be greeted by a map widget, featuring a searchable bar to search for specific locations and a toolbox to draw polygons.

The user will draw areas on the map from which they want to obtain statistical analysis and data. They can save these areas with a name attached to them. After saving, a card displaying the basic data of the drawn polygon will be shown in a side bar. This side bar will list all polygons created by the user.

The cards will display a 'Statistics' button upon saving, which will initially be disabled. This button will be enabled when the reports for the selected types of statistical analysis and data sources, as specified by the user, are ready. This functionality will be handled by an asynchronous worker.

After the report is ready, the button will be shown as enabled, and the user will have access to the full report on that area.

In the report sections, the user will have access to one or multiple reports, based on the data sources the user has subscribed to. This aspect will be discussed in more detail at a later stage, as most of the product development will be focused on this area where we can be the most creative. As we launch and grow, we will have the opportunity to add multiple data sources and types of analysis based on market feedback and research. For our Minimum Viable Product (MVP), we will initially include two data sources with basic analysis.

Geospatial data consists of spectral analysis of the soil and weather data, as well as dvi (difference vegetation index) and ndvi (normalized difference vegetation index) imaging and statistics. This type of data is particularly useful in agriculture and engineering research.

Geographical Statistics consists of the analysis and statistics of populations and socioeconomic databases. It is commonly used by researchers, students, journalists, and pretty much everybody who needs basic information from a particular area.

One important aspect of this product is that the data needs to be sourced from reputable sources. We will also need to implement an audit system to ensure our users can rely on the data we provide. This aspect of the project, coding the trust and reliability of the data, will be very fun as i have no idea how to do it yet.

Basic system design

I am not going to dive too much into system design as I believe it is currently pointless. Most of the crucial decisions regarding system design will be made along the way. For now, I will opt for a simple implementation, including a server to handle all requests and events, a worker to process asynchronous jobs that require more processing power, a storage solution, and a front-end application.

We will delve deep into each topic as development progresses. All repositories will be public, and I will document each step, showcasing all interactions with anyone who chooses to participate in any form in this project. A lot of things can and will go wrong and i am expecting to get the most out of it. Until the next update!

Catch me on twitter any time.

Photo by Clay Banks in Unsplash

How to use mongoDB change streams as a powerful event-driven engine

Caio Campos Borges Rosa — Wed, 12 Jul 2023 20:29:35 +0000

Before change streams became a feature of MongoDB, developers who wanted to track real-time changes in the database use to monitor oplog entries and track the changes in a specific collection based on timestamps. This process was often complex, and the mechanisms required for resuming and recovering reading were not particularly secure.

Change streams allow applications a direct interface in real time to deal with change in database collections, with powerful features to custom event-driven architectures.

Change streams are available in MongoDB when a replica set environment is configured. They rely on changes to the members as they reach consensus on a particular change in a majority of the members. This ensures the safety of data in a specific collection, especially in scenarios where failures may occur.

To utilize change streams in a local development environment, we configure a replica set using Docker Compose. For detailed instructions on this setup, you can refer to the guide written by @sibelius. We have invested considerable effort to ensure that all our developers can easily set up a local environment and work with the full range of features that MongoDB offers.

Setup

At Woovi we use change streams in a specific service called Publisher. This service is app that instantiates a subscriber for each collection in our Database from which we want to derive events.

import { companySubscriber } from './CompanySubscriber';
import { customerSubscriber } from './CustomerSubscriber';
import { userSubscriber } from './UserSubscriber';

export const setupSubscribers = () => {
  companySubscriber();
  userSubscriber();
  customerSubscriber();
};

Events

The role of the subscriber is to monitor a collection and generate a data event. This event is then passed to a handler, which processes the data according to our specific needs.

export const userSubscriber = async () => {
  const stream = User.watch([], { fullDocument: 'updateLookup' });

  stream.on('change', (data) => handleUserSubscriberEvent(data));
};

The stream output changes based on the event that occur on a particular collection. Mongodb has many change events as seen in the documentation.

The stream pipeline accepts multiple options, fullDocument: 'updateLookup' makes the event Update send the full document an not just updated fields as is default, each event has a particular stream payload you can use base on your application and may configurations to tailor as you need.

The first param is an array of pipeline options you can pass to modify the stream output.

const pipeline = [
  { $match: { 'fullDocument.username': 'alice' } },
  { $addFields: { newField: 'this is an added field!' } }
];
 const stream = User.watch(pipeline, { fullDocument: 'updateLookup' });
stream.on('change', (data) => handleUserSubscriberEvent(data));
});

Use cases

Finally, in our handleUserSubscriberEvent function, we utilize the data object to drive any event-driven service within our application environment. In our specific case, we use it to create and update indices in our Elastic Search Service, which serves as the core technology behind our internal search tools.


export const handleUserSubscriberEvent = (data) => {
  const dataPicked = {
    _id: data.fullDocument._id,
    name: data.fullDocument.name,
    email: data.fullDocument.email,
    cellphone: data.fullDocument.cellphone,
    taxID: data.fullDocument.taxID,
  };

  const obj = {
    data: dataPicked,
  };

  handleDocumentIndexing(ELASTICSEARCH_INDEXES.USER, obj);

You can use change streams as a powerful engine to event-driven applications such as:

Analytics Processing
Notifications
IoT integration with MQTT

The performance of mongoDB change streams allows Woovi to scale even more event-driven products as fast and secure as we can.

Woovi is a Startup that enables shoppers to pay as they like. To make this possible, Woovi provides instant payment solutions for merchants to accept orders.

If you want to work with us, we are hiring!

Observability with Elasticsearch Kibana and APM

Caio Campos Borges Rosa — Mon, 12 Jun 2023 12:14:32 +0000

Why at Woovi we invest in Observability?

At Woovi we focus on speed and innovation, in highly dynamic distributed systems, troubleshooting can be a challenging aspect, causing valuable time to be wasted on debugging. However, the concept of observability addresses this pain point by providing insights into data and processes, ultimately minimizing debugging costs and enabling more accurate future planning. Observability empowers users to gain visibility into the inner workings of their systems, helping them identify issues promptly and efficiently while freeing up time for the creation of new and innovative solutions.

Elastic Search

Elasticsearch is a versatile and scalable search and analytics engine designed to handle large volumes of data. It provides near-real-time search capabilities and supports complex querying, making it an ideal solution for a wide range of use cases. Built on top of the Apache Lucene library, Elasticsearch uses a distributed architecture to ensure high availability, fault tolerance, and scalability. It stores data in the form of JSON documents, allowing for efficient indexing and retrieval. It also offers features such as automatic sharding, replication, and distributed document storage, making it suitable for both small-scale applications and enterprise-level deployments.

APM

In Elasticsearch, the APM (Application Performance Monitoring) module provides insights into the performance of your applications and services that interact with Elasticsearch. APM in Elasticsearch works by instrumenting your application code and capturing detailed information about transactions, spans, and errors.

To use APM in Elasticsearch, you need to integrate the APM agent into your application. The agent is available in various programming languages and frameworks. Once integrated, the agent automatically collects performance data from your application and sends it to the Elasticsearch cluster.

Here we use event driven architecture, so our resources are divided in server and workers. One example of setup in a server for apm would be indexing the body of the request.



app.use(async (ctx, next) => {
  ctx.req.body = ctx.request.body;

  apm.setLabel('requestBody', ctx.request.rawBody);

  await next();
});

This is the most simple form of indexing processes with apm, you set a label and a value for something you like to monitor, that way you can have a well structure stream of data to search. In our case we label requests bodies for all our api endpoints that way we can have a data drive approach to debugging live. Out of the box we can have for a endpoint requests, latency, throughput and error statistics.

The APM module in Elasticsearch also supports distributed tracing, which allows you to follow a request's journey across multiple services and systems, providing insights into the end-to-end performance and metadata of your application.

Kibana

Kibana is a powerful data visualization and exploration tool that works in conjunction with Elasticsearch. It provides a user-friendly web interface for searching, analyzing, and visualizing data stored in Elasticsearch, making it easier to understand and derive insights from large datasets. Alongside with APM is a powerful tool to visualization and live monitoring.

A great approach for application data visualization is to understand the team needs and the data profile before developing labels. With this approach we can ensure not only that we have all data we need but don't lose a hand on storage, as our indexes take space and infra costs.

One of the problems kibana and apm solves here at Woovi is on the worker side of our application, how to debug events in multiple queues with multiple data profiles as fast as possible and with confidence on the data? We introduced a label called job.Originator. That label is set in each job created by our event emitters so that way we can see where the event was created in the first place and with hat inquire the data for that event in specific moments of our data lifetime.



import apm from 'elastic-apm-node';

export const createJobBull = async <T extends Record<string, any> = any>(
  name: string,
  data?: T,
  options?: JobOptions,
  originator?: string,
  queue: Queue = queues.DEFAULT,
): Promise<Job<T>> => {

  const span = apm.startSpan('createJobBull');

  if (span) {
    span.setLabel('jobName', name);
    if (data) {
      span.setLabel('jobData', JSON.stringify(data));
    }

    if (originator) {
      span.setLabel('JobOriginator', originator);
    }

    if (options) {
      span.setLabel('jobOptions', JSON.stringify(options));
    }

    span.end();
  }

  return queue.add(name, options);
};

Now we can search live and using data provided from the labels we set to query specific cases based on origin, data, resources and time.

If you want to work with us, we are hiring!