DEV Community: Ronilson Alves

How to automate deployment of applications to Kubernetes cluster using GitHub Actions

Ronilson Alves — Thu, 25 Jul 2024 19:20:31 +0000

Você pode ler este artigo em Português clicando aqui

While in last article of this series, I've teach you how to set up a free Kubernetes cluster running on Oracle Cloud's always free tier today you will learn how to automate the deployment of your applications from a GitHub repository to Kubernetes cluster using the GitHub Actions, building and publishing a Docker image to OCI Registry and empower you to move once and for all from any Vercel alike service.

We'll walk you through setting up policies in OCI Console, configuring your GitHub repository for automated builds and deployments, and ultimately deploying your Dockerized application to your OKE cluster.

Before You Begin

To follow this guide I presume that you already have:

A free Kubernetes cluster running on OCI (follow our previous article to set it up), also all the credentials necessary to interact with the cluster.
A basic understanding of Docker and Kubernetes concepts.
A GitHub account.

Also I would like to introduce you to the infrastructure architecture that will be used in this guide.

Let's Get Started!

1. Securing Your OKE Cluster with Network Security Groups

OCI recommends managing security rules for your OKE cluster using Network Security Groups (NSGs). These groups allow you to define granular access controls for your applications running within the cluster.

1.1 Configuring policies to allow cluster manage NSGs.

In your OCI console, navigate to Identity & Security > Policies.

Click Create Policy.
Name your policy (e.g., "OKE-Security") and provide a description.
Select the Show manual editor option.
Add the following policy statements to the editor:

ALLOW any-user to manage network-security-groups in compartment <compartment-name> where request.principal.type = 'cluster'
ALLOW any-user to manage vcns in compartment <compartment-name> where request.principal.type = 'cluster'

Replace with the name of your OKE cluster's compartment (if you have created one).

If your cluster was created at the root compartment, use compartment id and the tenant OCID instead. ... in compartment id <tenant-OCID>

1.2 Creating a default NSG for NLB backends

The next step is create a NSG for managing the nlbs' backends. In your OCI console navigate to Virtual Cloud Networks > select the vcn where OKE is running.

In resources, select Network Security Groups.
Click Create Network Security Group.
Name your new NSG (e.g., "nlb-nsg-backend") and click Next.
In Security Rules don't add any rules and click Create.

1.3 Attaching NSG created to node workers.

To have effect and oci-cloud-controller-manager make his magic orchestrating security rules in cluster's VCN and exposing correctly to the internet the Kubernetes LoadBalancer services type we need to attach the created NSG to all the node workers.

In OCI Console go to Developer Services > Containers & Artifacts > Kubernetes Clusters (OKE).

Select the cluster previously created.
In resources, select node pools and click in the pool name.
Click node name, a page with instance information should be shown.
In VNICS, seek for Network security groups and click edit.
Select the previous created NSG, repeat the process to all node instances.

2. Setting Up Your GitHub Repository

To Dockerize our application and create a valid workflow for GitHub Actions we need to set up some things in our repository.

Create a new directory in your repository for your workflow files:

.github/workflows

Create a Dockerfile within this directory to define your application's build instructions.
Example Dockerfile for a Go Lang application:

FROM golang:1.22.5-alpine3.19 AS build
WORKDIR /app
COPY go.mod ./
RUN go mod download
COPY . ./
RUN CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -o hello main.go

FROM gcr.io/distroless/static-debian11:nonroot
COPY --from=build /app/hello /hello
USER nonroot:nonroot
EXPOSE 8080
ENTRYPOINT ["/hello"]

Create a directory for deployments manifests (optional):

/kubernetes

2.1 Preparing an YAML manifest to deployment:

Imagine that your repo hosts the code of a Go Lang application, let's prepare a YAML file to describe the deployment and service that will be deployed to Kubernetes cluster.

Example Deployment YAML manifest file:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: go-hello-world
  labels:
    app: go-hello-world
spec:
  replicas: 2
  selector:
    matchLabels:
      app: go-hello-world
  template:
    metadata:
      labels:
        app: go-hello-world
    spec:
      containers:
      - name: go-hello-world
        image: $IMAGE_NAME
        imagePullPolicy: Always
        ports:
          - containerPort: 8080
            protocol: TCP
      imagePullSecrets:
        - name: $DOCKER_REGISTRY_SECRET_NAME

The deployment kind defines the application go-hello-world details, including pod configs, see we're referring to an environment variable to set image and a docker registry secret $IMAGE_NAME and $DOCKER_REGISTRY_SECRET_NAME that will be created and set dynamically when running the workflow action that will be created soon.

Example Service YAML manifest

---
apiVersion: v1
kind: Service
metadata:
  name: go-hello-world-lb
  labels:
    app: go-hello-world
  annotations:
    oci.oraclecloud.com/load-balancer-type: "nlb"
    oci-network-load-balancer.oraclecloud.com/backend-policy: "THREE_TUPLE"
    oci.oraclecloud.com/security-rule-management-mode: "NSG"
    oci-network-load-balancer.oraclecloud.com/security-list-management-mode:  "None"
    oci.oraclecloud.com/oci-backend-network-security-group: $OCI_NETWORK_SG_BACKEND
spec:
  type: LoadBalancer
  externalTrafficPolicy: Local
  ports:
    - name: http
      port: 80
      targetPort: 8080
  selector:
    app: go-hello-world

The service kind exposes the app go-hello-world to the internet through a LoadBalancer service type, listening on port 80 and redirecting the traffic to 8080 port of the pods. As seen, there's specific OCI annotations used to define the type of load balancer (NLB) and handle the network security policies, in details, the oci.oraclecloud.com/oci-backend-network-security-group annotation refers to an another environment variable to set the OCID of NSG for Network Load Balancer created and attached to the nodes instances earlier in this guide, again, when running the workflow action we will set dynamically this data.

Example repository structure after all these changes:

.dockerignore
.github
   |-- workflows
   |   |-- main.yml
.gitignore
Dockerfile
go.mod
kubernetes
   |-- main.yml
main.go

3. Configuring GitHub Actions Secrets

To automate deployments, we'll use GitHub Actions secrets to store sensitive information like your OCI credentials and cluster details.

Navigate to your repository's Settings > Secrets & variables > Actions.
Create the following repository's secrets, replacing the placeholders with your actual values:
- OKE_CLUSTER_OCID: The cluster id of OKE cluster, available at cluster details page.
- OCI_CLI_REGION: Your OCI region (e.g., sa-saopaulo-1).
- OCI_CLI_TENANCY: Your OCI tenancy OCID.
- OCI_CLI_USER: Your OCI user OCID.
- OCI_CLI_FINGERPRINT: Your OCI user's public key fingerprint.
- OCI_CLI_KEY_CONTENT: Your OCI user's private key content.
- OCIR_AUTH_TOKEN: Your OCI Registry authentication token (generated in your OCI Console).
- OCIR_REGISTRY: Your OCI Registry domain (e.g., ocir.oraclecloud.com).
- OCIR_USERNAME: Your OCI Registry username (namespace + email address, e.g., sbataw3dd/jhon@acme.com).
- OCIR_USERNAME_EMAIL: The email from username
- OCIR_REPO_NAME: The name of your OCI Registry repository.
- OCI_NETWORK_SG_BACKEND: The OCID of the Network Security Group attached to your OKE cluster's worker nodes.
- OCI_TENANCY_NAMESPACE: The namespace generated automatically for your OCI tenancy's Object Storage.

4. Creating Your GitHub Actions Workflow

Create a file named main.yml within your workflows directory. This file will define the steps for your automated deployment process. The file we will create is available in example GitHub repository for this article, link at the end of the article.

Before we proceed, we have to back to OCI Console and certify that is possible to create a new container registry repository at first push to root compartment.

Navigate to Developer Services > Containers & Artifacts > Container Registry

Click Settings, check Create repository on first push in root compartment option and Save changes.

Fine, let's finally to GitHub Actions Workflow

name: Build, push and deploy example

on:
  push:
    branches:
      - main
  pull_request:
    branches:
      - main
jobs:
  build-and-publish-docker-image:
    name: Build and publish Docker image
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v4
      - name: Set up QEMU
        uses: docker/setup-qemu-action@v3
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3
      - name: Login to Oracle Cloud Registry
        uses: docker/login-action@v3
        with:
          registry: ${{ secrets.OCIR_REGISTRY }}
          username: ${{ secrets.OCIR_USERNAME }}
          password: ${{ secrets.OCIR_AUTH_TOKEN }}
      - name: Build Docker image
        uses: docker/build-push-action@v6
        with:
          platforms: linux/arm64
          context: .
          file: Dockerfile
          tags: ${{ secrets.OCIR_REGISTRY }}/${{ secrets.OCI_TENANCY_NAMESPACE }}/${{ secrets.OCIR_REPO_NAME }}:latest
          push: false
          load: true
          cache-from: type=gha
          cache-to: type=gha,mode=max
      - name: Push Docker Image to Oracle Cloud Registry
        run: |
          docker push ${{ secrets.OCIR_REGISTRY }}/${{ secrets.OCI_TENANCY_NAMESPACE }}/${{ secrets.OCIR_REPO_NAME }}:latest

  deploy-to-oke:
    needs: build-and-publish-docker-image
    name: Deploy container image to an OKE Cluster
    runs-on: ubuntu-latest 
    env:
      OCI_CLI_USER: ${{ secrets.OCI_CLI_USER }}
      OCI_CLI_TENANCY: ${{ secrets.OCI_CLI_TENANCY }}
      OCI_CLI_FINGERPRINT: ${{ secrets.OCI_CLI_FINGERPRINT }}
      OCI_CLI_KEY_CONTENT: ${{ secrets.OCI_CLI_KEY_CONTENT }}
      OCI_CLI_REGION: ${{ secrets.OCI_CLI_REGION }}
      OCI_NETWORK_SG_BACKEND: ${{ secrets.OCI_NETWORK_SG_BACKEND }}
      IMAGE_NAME: ${{ secrets.OCIR_REGISTRY }}/${{ secrets.OCI_TENANCY_NAMESPACE }}/${{ secrets.OCIR_REPO_NAME }}:latest
      DOCKER_REGISTRY_SECRET_NAME: 'name-of-your-secret'
      NAMESPACE: 'name-of-desired-kubernetes-namespace'
    steps:
      - name: Configure Kubectl
        uses: oracle-actions/configure-kubectl-oke@v1.5.0
        with:
          cluster: ${{ secrets.OKE_CLUSTER_OCID }}
      - name: Checkout
        uses: actions/checkout@v4
      - name: Create namespace if not exists
        run: |
          kubectl create namespace ${{ env.NAMESPACE }} || echo "namespace already exists"
      - name: Create Docker registry secret
        run: | 
          kubectl -n preview create secret docker-registry ${{ env.DOCKER_REGISTRY_SECRET_NAME }} \
          --docker-server=${{ secrets.OCIR_REGISTRY}} \
          --docker-username=${{ secrets.OCIR_USERNAME}} \
          --docker-password='${{ secrets.OCIR_AUTH_TOKEN }}' \
          --docker-email=${{ secrets.OCIR_USERNAME_EMAIL}} || echo "secret already exists"
      - name: Deploy to Kubernetes Cluster
        run: |
          envsubst < kubernetes/main.yml | kubectl apply -f - -n ${{ env.NAMESPACE }}

This workflow is triggered on push and pull request events to the main branch. Let's dive deeper in the workflow and understand better the jobs and steps of workflow:

build-and-publish-docker-image Job

This job is responsible for building and publishing a Docker image.

Checkout: This step checks out the code from the GitHub repository inside the runner environment. It uses the actions/checkout@v4 official GitHub action.
Set up QEMU: This step sets up QEMU, a virtual machine emulator, for building and testing Docker images. It uses the docker/setup-qemu-action@v3 official Docker action.
Set up Docker Buildx: This step sets up Docker Buildx, a tool for building and pushing Docker images. It uses the docker/setup-buildx-action@v3 official Docker action.
Login to Oracle Cloud Registry: This step logs in to the Oracle Cloud Registry using the provided credentials (username, password, and registry URL). It uses the docker/login-action@v3 official Docker action.
Build Docker image: This step builds a Docker image using the docker/build-push-action@v6 official Docker action. The build context is the current directory (.), the Dockerfile is used as the build script, and the image is tagged with the Oracle Cloud Registry URL and the latest tag. The push option is set to false to prevent the image from being pushed to the registry immediately, for some weird reason, when pushing direct to OCIR the image created it's split into layers.
Push Docker Image to Oracle Cloud Registry: This step pushes the built Docker image to the Oracle Cloud Registry using the docker push command.

deploy-to-oke Job

This job is responsible for deploying the built Docker image to an Oracle Kubernetes Engine (OKE) cluster.

Needs: This directive specifies that the build-and-publish-docker-image job must complete successfully before this job can start.
Configure Kubectl: This step configures the kubectl command-line tool to connect to the specified OKE cluster. It uses the oracle-actions/configure-kubectl-oke@v1.5.0 action.
Checkout: This step checks out the code from the GitHub repository inside the runner environment. It uses the actions/checkout@v4 official Docker action.
Create namespace if not exists: This step creates a Kubernetes namespace if it does not already exist. It uses the kubectl create namespace command.
Create Docker registry secret: This step creates a Kubernetes secret to store the Oracle Cloud Registry credentials. It uses the kubectl create secret command.
Deploy to Kubernetes Cluster: This step deploys the Docker image to the OKE cluster using the kubectl apply command. The deployment configuration is read from a file named kubernetes/main.yml and the envsubst command is used to replace environment variables in the file.

If all occurs well in the next push or pull request to the main branch of our repository will trigger the action and it will build and publish the image to OCI private registry and deploy the Dockerized application to Kubernetes cluster.

The OKE itself will provision an OCI NLB and will set up all network related config and security rules. At the end, you will have an NLB with a public IPV4 that you can aim for a domain, also it's possible to set up a reserved IP in Kubernetes service type manifest, please for more details of how to, read OCI documentation.

GitHub Action executed successfully:

Pods running:

NAME                              READY   STATUS    RESTARTS   AGE
go-hello-world-6cbdd98797-hs7lv   1/1     Running   0          7m10s
go-hello-world-6cbdd98797-t4d5f   1/1     Running   0          7m10s

NLB provisioned by OKE Cluster:

Conclusion

This is a simple guide to automate the deployment of your applications to Kubernetes cluster and empower you to move from any PaaS with Docker approach to a public cloud. You can include job workflows to test your code before deploying it to a Kubernetes cluster, also you can use GitHub Actions workflows to improve your entire CI/CD cycle managing multiple environments (e.g., production, preview, staging).

I hope this article has been useful for you, let me know if you have any insights or comments about it. See you!

Links & references

How to deploy a free Kubernetes Cluster with Oracle Cloud Always free tier

Ronilson Alves — Tue, 16 Jul 2024 17:57:05 +0000

Você pode ler este artigo em Português clicando aqui.

The recent changes in Vercel's billing system have prompted many developers to seek alternative platforms that offer cost-effective solutions without compromising on capabilities. Oracle Cloud Infrastructure (OCI) presents itself as a viable alternative with its Always Free tier, which provides users with a robust set of services that can cater to various computing needs, this series of articles will explore the process of migrating from Vercel – or another PaaS to Oracle Cloud, offering a comprehensive guide for those considering the switch.

Oracle Cloud Always Free tier

Oracle Cloud's Always Free tier is designed to give developers a hands-on experience with Oracle's cloud services. With $300 in credits available for the first 30 days and a selection of services that remain free indefinitely, it's an attractive option for those starting out or looking to migrate from another platform.

The Ampere A1 Compute Instances

One of the standout offerings in the Always Free tier is the Ampere A1 Compute instances. These virtual machines (VMs) are based on the ARM architecture and are known for their efficiency and scalability. Users can deploy up to four VMs, harnessing up to 24GB of memory, which is more than sufficient for a range of applications, including development, testing, and even production workloads.

Step-by-Step Guide to Deploying a free OKE Cluster

To follow this guide you must have an Oracle Cloud account. In some regions you probably need to make an upgrade in your account to reach the free ARM instances due to low availability in Always free tier, I, for e.g., have tried for weeks deploy the ARM Instances in regions sa-saopaulo-1 and sa-vinhedo-1 without success using a free account, just succeed after I upgrade my account and the instances became available.

So, logged in your account, in console go to Developer Services > Containers & Artifacts > OKE.

After, just click on Create cluster button and the modal will be shown, if you don't have any contact with infra or It's your first contact with public clouds or even OCI itself, select the Quick create option.

So, OCI will provide the necessary resources to orchestrate the network of our cluster, creating a virtual network, public and private subnets and also some related services to empower the communication between Kubernetes panel and nodes and also external access.

On creation screen, fill the cluster name, select the compartment in your OCI tenancy and the Kubernetes version should be used in your cluster. Also, select if you want or not to expose to internet your Kubernetes API endpoint and Kubernetes nodes, in this article, we will use private nodes, but feel free to choose your preferred configuration.

To allow us to use the free A1 Ampere instances in our cluster, select the node type managed as seen above.

The next step in Image & shape, select the VM.Standard.A1.Flex VM inside Node shape. Following this guide, we use 2 nodes, so set the OCPU and memory to 2 and 12 GB respectively and Node count to 2 as you can see in the below image. You can configure as you prefer, just remember to not overtake the Always free tier limit of 4 OCPU and 24GB of memory. Mainly if you have upgraded your account to a paid account to avoid surprise charges.

Click in Show advanced options if you want to specify a custom boot volume size and/or upload a public SSH key, if not, just click in Next button.

After, a screen with all resources that will be created will be shown, and before you click on Create cluster button, check the option to create a basic cluster instead of an enhanced cluster at the end of the page. If you create an enhanced cluster, you should pay by this resource and most important, the basic cluster is FREE!

Few minutes later OCI will finish the OKE provisioning process and you are able to access your cluster using Cloud Shell Access or using OCI CLI. Remember to install OCI CLI if you want to access your cluster locally.

Conclusion

Migrating from a PaaS like Vercel to Oracle Cloud may seem like a challenging step, but with the always free tier and specifically the Ampere A1 instances, this transition becomes more accessible and attractive to developers.

This step-by-step guide demonstrated how to create a free OKE cluster in OCI, opening doors for deploying applications in a robust and scalable environment.

By exploring the capabilities of OCI and Kubernetes, developers can discover new possibilities to optimize their workflows and build more efficient software solutions.

In the next article of this series, we will delve deeper into the automation process with GitHub Actions and how to setup the created OKE Cluster to these automation. Don't forget to let your comment and share your insights about this guide.

Understanding Semantic Commits: What They Are, Why, and How to Use Them

Ronilson Alves — Tue, 05 Sep 2023 02:07:08 +0000

Introduction

In software development, teamwork is crucial to ensure an efficient workflow and the delivery of high-quality projects. One of the practices that aids in this process is the use of semantic commits. In this article, we will explore the concept of semantic commits, understand why they are important, and learn how to effectively use them.

1. What are Semantic Commits?

Semantic commits are a naming convention for commit messages in a version control repository. They follow a predefined pattern and provide clear and concise information about the changes made at a given time. These messages are structured and contain information about the type of change, the scope, and a brief description.

2. Why Use Semantic Commits?

2.1 Clarity and Understanding

Using semantic commits makes it easier to understand the changes made in a particular commit. Well-structured and descriptive messages allow team members to quickly grasp the modifications made and the reasoning behind them.

2.2 Detailed Change History

Semantic commits also contribute to a more detailed change history. By following a consistent naming convention, it is possible to obtain a clear and organized record of the modifications made over time.

2.3 Facilitates Code Review

When using semantic commits, code review becomes more efficient. Well-structured commit messages enable reviewers to easily comprehend the proposed changes, saving time and effort.

3. How to Use Semantic Commits?

3.1 Message Convention

To use semantic commits, it is important to follow a message convention. One popular convention is the one specified by AngularJS, which consists of three main parts: type, scope, and description. Some of the most common commit types are:

feat: for adding a new feature.
fix: for fixing a bug.
docs: for documentation changes.
chore: for project maintenance tasks.
test: for adding or modifying tests.

3.2 Example of a Semantic Commit Message

An example of a semantic commit message would be:

feat: Add login functionality

Adds login functionality to allow users to access their accounts in the application.

Conclusion

Semantic commits are a powerful practice to improve collaboration and version control in software development teams. By using well-structured commit messages, you can achieve clarity, detailed history, and facilitate code review. Start using semantic commits in your next project and experience the benefits they bring.

FAQs

Why should I adopt semantic commits?
Semantic commits provide clarity, detailed history, and facilitate code review, improving teamwork workflow.
Are there other naming conventions for semantic commits?
Yes, besides the AngularJS convention, there are other popular conventions such as Conventional Commits.
How can I incorporate semantic commits into my existing workflow?
You can start using semantic commits by defining a naming convention and guiding your team on how to use them.
Are semantic commits only applicable to large-scale projects?
No, semantic commits can be beneficial in projects of any size as they improve communication and clarity of changes made.
Are there any tools that assist in adopting semantic commits?
Yes, there are tools like commitizen and commitlint that help adhere to semantic commit conventions.

Building a Ethereum wallet watcher using concurrent programming in GoLang

Ronilson Alves — Mon, 03 Jul 2023 03:00:00 +0000

Cover photo by Mika Baumeister / Unsplash

In my last article I talked about how simple it is to build applications that make use of concurrent programming using the Go language, we saw how simple it is to implement and how efficient it can be, thanks to its construction that provides developers with powerful tools to take full advantage of the potential of concurrent programming. In this article we will take a practical step-by-step look at how to build an Ethereum wallet watcher taking advantage of this strength of GoLang.

Context

Web3 since I returned to my programming studies has always been a topic I am very interested in, I am an enthusiast of the blockchain technology behind most of the products developed and which has been adopted in more traditional solutions so to speak recently. Some wallets have their private keys publicly exposed, most of them being made available by development kits where you can run a test node in a local environment. Some users, by carelessness and lack of attention, end up sending values to the address of this wallet on the main network (in production so to speak).

Goal

We will build a service in GoLang to look at Ethereum wallets and if there are any incoming balance transactions, we will try to perform an outgoing transaction to get those values.

To further exemplify the use of concurrent programming we will also provide an API for querying information from an Ethereum address.

In summary, we will run our wallet watcher service side by side with a rest API. Let's go?

Before we begin

The source code of this project is on my GitHub, you can clone it using git clone and running it on your machine, don't forget to create an .env file in the root of the project following the .env.example

git clone https://github.com/ronilsonalves/go-wallet-watcher.git

If you also want to build the Rest API for queries, you will need to have an API key from Etherscan.io, for this you will need to have an account and request a free key at https://etherscan.io/myapikey

Getting started

Where we will save our project and start it, for this, in the terminal we need to type:

go mod init 'project-name'

Before proceeding, let's install the go-ethereum and godotenv modules that will help us build our service:

go get github.com/ethereum/go-ethereum
go get github.com/joho/godotenv

Now, we must create a directory called internal where we will organize the internal files of our project, inside it we will must create another directory folder which will be our domain package and finally, we will create our file wallet.go which will contain a struct that will represent an Ethereum wallet:

package domain

type Wallet struct {
    Address      string        `json:"address"`
    SecretKey    string        `json:"secret-key,omitempty"`
    Balance      float64       `json:"balance"`
    Transactions []Transaction `json:"transactions,omitempty"`
}

Also, we must create an struct to represent a transaction, we will use it in further:

package domain

type Transaction struct {
    BlockNumber       string `json:"blockNumber,omitempty"`
    TimeStamp         string `json:"timeStamp,omitempty"`
    Hash              string `json:"hash,omitempty"`
    Nonce             string `json:"nonce,omitempty"`
    BlockHash         string `json:"blockHash,omitempty"`
    TransactionIndex  string `json:"transactionIndex,omitempty"`
    From              string `json:"from,omitempty"`
    To                string `json:"to,omitempty"`
    Value             string `json:"value,omitempty"`
    Gas               string `json:"gas,omitempty"`
    GasPrice          string `json:"gasPrice,omitempty"`
    IsError           string `json:"isError,omitempty"`
    TxreceiptStatus   string `json:"txreceipt_status,omitempty"`
    Input             string `json:"input,omitempty"`
    ContractAddress   string `json:"contractAddress,omitempty"`
    CumulativeGasUsed string `json:"cumulativeGasUsed,omitempty"`
    GasUsed           string `json:"gasUsed,omitempty"`
    Confirmations     string `json:"confirmations,omitempty"`
    MethodId          string `json:"methodId,omitempty"`
    FunctionName      string `json:"functionName,omitempty"`
}

Creating our wallet watcher using goroutines

Still inside internal directory we will create another package named 'watcher', inside it we will create our service.go file where we will implement our watcher, fist we will create a function responsible for starting our service:

// StartWatcherService load from environment the data and start running goroutines to perform wallet watcher service.
func StartWatcherService() {

    err := godotenv.Load()
    if err != nil {
        log.Fatalln("Error loading .env file", err.Error())
    }

    var wfe [20]domain.Wallet
    var wallets []domain.Wallet

    for index := range wfe {
        wallet := domain.Wallet{
            Address:   os.Getenv("WATCHER_WALLET" + strconv.Itoa(index+1)),
            SecretKey: os.Getenv("WATCHER_SECRET" + strconv.Itoa(index+1)),
        }
        wallets = append(wallets, wallet)
    }
    // contains filtered fields or functions
}

In the above code snippet we load our environment variables where we stored data that should not be exposed, in this example we have loaded 20 wallets and their respective private keys using the for range.

Still in our service.go, we will create an synchronization group for our go routines that will be created:

// StartWatcherService load from environment the data and start running go routines to perform wallet watcher service.
func StartWatcherService() {

    // contains filtered fields or functions

    // Create a wait group to synchronize go routines
    var wg sync.WaitGroup
    wg.Add(len(wallets))

    // contains filtered fields or functions
}

Following, we will create our go routines:

// StartWatcherService load from environment the data and start running go routines to perform wallet watcher service.
func StartWatcherService() {
    // contains filtered fields or functions
    // Start a go routine for each wallet
    for _, wallet := range wallets {
        go func(wallet domain.Wallet) {
            // Connect to the Ethereum client
            client, err := rpc.Dial(os.Getenv("WATCHER_RPC_ADDRESS"))
            if err != nil {
                log.Printf("Failed to connect to the RPC client for address %s: %v \n Trying fallback rpc server...", wallet.Address.Hex(), err)
            }
            client, err = rpc.Dial(os.Getenv("WATCHER_RPC_FALLBACK_ADDRESS"))
            if err != nil {
                log.Printf("Failed to connect to the Ethereum client for address %s: %v", wallet.Address.Hex(), err)
                wg.Done()
                return
            }

            // Create an instance of the Ethereum client
            ethClient := ethclient.NewClient(client)

            for {
                // Get the balance of the address
                balance, err := ethClient.BalanceAt(context.Background(), common.HexToAddress(wallet.Address), nil)
                if err != nil {
                    log.Printf("Failed to get balance for address %s: %v", wallet.Address.Hex(), err)
                    continue
                }

                balanceInEther := new(big.Float).Quo(new(big.Float).SetInt(balance), big.NewFloat(1e18))

                log.Printf("Balance for address %s: %.16f ETH", wallet.Address.Hex(), balanceInEther)


                // if the wallet has a balance superior to 0.0005 ETH, we are sending the balance to another wallet
                if balanceInEther.Cmp(big.NewFloat(0.0005)) > 0 {
                    sendBalanceToAnotherWallet(common.HexToAddress(wallet.Address), balance, wallet.SecretKey)
                }

                time.Sleep(300 * time.Millisecond) // Wait for a while before checking for the next block
            }
        }(wallet)
    }
        // Wait for all go routines to finish
    wg.Wait()
}

Finally, we will create our function responsible for generating and signing a transaction that will send the wallets' balance to another wallet:

// sendBalanceToAnotherWallet when find some values in any wallet perform a SendTransaction(ctx context.Context,
// tx *types.Transaction) function
func sendBalanceToAnotherWallet(fromAddress common.Address, balance *big.Int, privateKeyHex string) {
    toAddress := common.HexToAddress(os.Getenv("WATCHER_DEST_ADDRESS"))
    chainID := big.NewInt(1)

    // Connect to the Ethereum client
    client, err := rpc.Dial(os.Getenv("WATCHER_RPC_ADDRESS"))
    if err != nil {
        log.Printf("Failed to connect to the Ethereum client: %v...", err)
    }

    ethClient := ethclient.NewClient(client)

    // Load the private key
    privateKey, err := crypto.HexToECDSA(privateKeyHex[2:])
    if err != nil {
        log.Fatalf("Failed to load private key: %v", err)
    }

    // Get the current nonce for the fromAddress
    nonce, err := ethClient.PendingNonceAt(context.Background(), fromAddress)
    if err != nil {
        log.Printf("Failed to retrieve nonce: %v", err)
    }

    // Create a new transaction
    gasLimit := uint64(21000) // Here we define the transaction gas tax limit based up on his type.
    gasPrice, err := ethClient.SuggestGasPrice(context.Background())
    if err != nil {
        log.Printf("Failed to retrieve gas price: %v", err)
    }


    tx := types.NewTx(&types.LegacyTx{
        Nonce:    nonce,
        GasPrice: gasPrice,
        Gas:      gasLimit,
        To:       &toAddress,
        Value:    new(big.Int).Sub(balance, new(big.Int).Mul(gasPrice, big.NewInt(int64(gasLimit)))),
        Data:     nil,
    })
    valueInEther := new(big.Float).Quo(new(big.Float).SetInt(tx.Value()), big.NewFloat(1e18))
    if valueInEther.Cmp(big.NewFloat(0)) < 0 {
        log.Println("ERROR: Insufficient funds to make transfer")
    }

    // Sign the transaction
    signedTx, err := types.SignTx(tx, types.NewEIP155Signer(chainID), privateKey)
    if err != nil {
        log.Printf("Failed to sign transaction: %v", err)
    }

    // Send the signed transaction
    err = ethClient.SendTransaction(context.Background(), signedTx)
    if err != nil {
        log.Printf("Failed to send transaction: %v", err)
    } else {
        log.Printf("Transaction sent: %s", signedTx.Hash().Hex())
    }
}

Our wallet watcher service is ready, our watcher/service.go should be like:

At this point, if we don't want to create an API Rest, we need just call our StartWatcherService() into our main.go

func main() {
    // filtered fields or functions

    // Start our watcher
    go watcher.StartWatcherService()

    // Wait for the server and the watcher service to finish
    select {}
}

Exposing a Rest API to queries

We will use the Gin Web Framework to build a Rest API where we will expose an endpoint to querying a wallet's balance and recent transactions from an address. To do this we need to add o gin-gonic module to our project:

go get github.com/gin-gonic/gin

Creating a service to our 'wallet' package

Now, inside internal we will create a 'wallet' package, in this package we will create a service.go file, this is where we will make calls to the Etherscan.io API to balance and transactions queries:

type Service interface {
    GetWalletBalanceByAddress(address string) (domain.Wallet, error)
    GetTransactionsByAddress(address, page, size string) (domain.Wallet, error)
}

type service struct{}

// NewService creates a new instance of the Wallet Service.
func NewService() Service {
    return &service{}
}

First we will create a method to get informations from address that we will receive as param (we will see soon the gin handler function):

// GetWalletBalanceByAddress retrieves the wallet balance for the given address
func (s service) GetWalletBalanceByAddress(address string) (domain.Wallet, error) {

}

Inside this we will get the Etherscan.io APIKey from our environment:

// GetWalletBalanceByAddress retrieves the wallet balance for the given address
func (s service) GetWalletBalanceByAddress(address string) (domain.Wallet, error) {
    // Retrieves Etherscan.io API Key from environment
    apiKey := os.Getenv("WATCHER_ETHERSCAN_API")
    url := fmt.Sprintf(fmt.Sprintf("https://api.etherscan.io/api?module=account&action=balance&address=%s&tag=latest&apikey=%s", address, apiKey))
    // Contains filtered fields or functions
}

Following, we will make a HTTP GET cal to Etherscan API and read the response content:

// GetWalletBalanceByAddress retrieves the wallet balance for the given address
func (s service) GetWalletBalanceByAddress(address string) (domain.Wallet, error) {

    // Contains filtered fields or functions

    // Send GET request to the Etherscan API
    response, err := http.Get(url)
    if err != nil {
        log.Printf("Failed to make Etherscan API request: %v", err)
        return domain.Wallet{}, err
    }
    defer response.Body.Close()

    // Read the response body
    body, err := io.ReadAll(response.Body)
    if err != nil {
        log.Printf("Failed to read response body: %v", err)
        return domain.Wallet{}, err
    }

    // Creates a struct to represent etherscan API response
    var result struct {
        Status  string `json:"status"`
        Message string `json:"message"`
        Result  string `json:"result"`
    }

    // Contains filtered fields or functions
}

Finally we will parse the Etherscan.io API response and structure it according with our struct, also we need to make some validations and return the data:

// GetWalletBalanceByAddress retrieves the wallet balance for the given address
func (s service) GetWalletBalanceByAddress(address string) (domain.Wallet, error) {

    // Contains filtered fields or functions

    // Parse the JSON response
    err = json.Unmarshal(body, &result)
    if err != nil {
        log.Printf("Failed to parse JSON response: %v", err)
        return domain.Wallet{}, err
    }

    if result.Status != "1" {
        log.Printf("API returned error: %s", result.Message)
        return domain.Wallet{}, fmt.Errorf("API error: %s", result.Message)
    }

    wbBigInt := new(big.Int)
    wbBigInt, ok := wbBigInt.SetString(result.Result, 10)
    if !ok {
        log.Println("Failed to parse string to BigInt")
        return domain.Wallet{}, fmt.Errorf("failed to parse string into BigInt. result.Result value: %s", result.Result)
    }

    wb := new(big.Float).Quo(new(big.Float).SetInt(wbBigInt), big.NewFloat(1e18))
    v, _ := strconv.ParseFloat(wb.String(), 64)

    return domain.Wallet{
        Address: address,
        Balance: v,
    }, nil
}

We already have our method to get balance information from a wallet address, now let's create another method in our wallet/service.go file to show the transactions, the logic will be the same as the previous method, the difference will be in how we map the Etherscan.io API response and how we will build the URL endpoint to make GET request, because we will have as parameters the page and the number of items per page beyond the address:

// GetTransactionsByAddress retrieves the wallet balance and last transactions for the given address paggeable
func (s service) GetTransactionsByAddress(address, page, size string) (domain.Wallet, error) {
    // Call to GetWalletBalanceByAddress to mount wallet with current balance.
    wallet, _ := s.GetWalletBalanceByAddress(address)
    apiKey := os.Getenv("WATCHER_ETHERSCAN_API")
    url := fmt.Sprintf("https://api.etherscan.io/api?module=account&action=txlist&address=%s&startblock=0&endblock=99999999&page=%s&offset=%s&sort=desc&apikey=%s", address, page, size, apiKey)

    // Contains filtered fields or functions

    // Parse the JSON response
    var transactions struct {
        Status  string               `json:"status"`
        Message string               `json:"message"`
        Result  []domain.Transaction `json:"result"`
    }

    // Add transactions to wallet struct
    wallet.Transactions = append(wallet.Transactions, transactions.Result...)

    return wallet, nil
}

We finished our wallet/service.go and the entire file should be like the gist bellow:

Creating the gin handler functions to expose our API

Following we will create our gin handlerFunc to interact with our wallet/service.go and expose the necessary endpoints to querying wallet's balance and transactions from an Ethereum wallet.

We will create a directory in the our project root and mame it as cmd (here we will put our main.go and the handler package from our API, the directory structure must be like that:

.env.example
cmd
   |-- server
   |   |-- handler
   |   |   |-- wallet.go
   |   |-- main.go
internal
   |-- domain
   |   |-- transaction.go
   |   |-- wallet.go
   |-- wallet
   |   |-- dto.go
   |   |-- service.go
   |-- watcher
   |   |-- service.go
pkg
   |-- web
   |   |-- response.go

Finally we will create our handler package, inside it we will create a wallet.go file:

package handler

type walletHandler struct {
    s wallet.Service
}

// NewWalletHandler creates a new instance of the Wallet Handler.
func NewWalletHandler(s wallet.Service) *walletHandler {
    return &walletHandler{s: s}
}

Inside this file we will create two handler functions: GetWalletByAddress() and GetTransactionsByAddress() to show wallet's balance and transactions:

// GetWalletByAddress get wallet info balance from a given address
func (h *walletHandler) GetWalletByAddress() gin.HandlerFunc {
    return func(ctx *gin.Context) {
        ap := ctx.Param("address")
        w, err := h.s.GetWalletBalanceByAddress(ap)
        if err != nil {
            web.BadResponse(ctx, http.StatusBadRequest, "error", err.Error())
            return
        }
        web.OKResponse(ctx, http.StatusOK, w)
    }
}

// GetTransactionsByAddress retrieves up to 10000 transactions by given adrress in a paggeable response
func (h *walletHandler) GetTransactionsByAddress() gin.HandlerFunc {
    return func(ctx *gin.Context) {
        address := ctx.Param("address")
        page := ctx.Query("page")
        size := ctx.Query("pageSize")

        if len(page) == 0 {
            page = "1"
        }
        if len(size) == 0 {
            size = "10"
        }

        if _, err := strconv.Atoi(page); err != nil {
            web.BadResponse(ctx, http.StatusBadRequest, "error", fmt.Sprintf("Invalid page param. Verify page value: %s", page))
            return
        }

        if _, err := strconv.Atoi(size); err != nil {
            web.BadResponse(ctx, http.StatusBadRequest, "error", fmt.Sprintf("Invalid pageSize param. Verify pageSize value: %s", size))
            return
        }

        response, err := h.s.GetTransactionsByAddress(address, page, size)
        if err != nil {
            web.BadResponse(ctx, http.StatusBadRequest, "error", err.Error())
            return
        }

        var pageableResponse struct {
            Page  string      `json:"page"`
            Items string      `json:"items"`
            Data  interface{} `json:"data"`
        }

        pageableResponse.Page = page
        pageableResponse.Items = size
        pageableResponse.Data = response

        web.OKResponse(ctx, http.StatusOK, pageableResponse)
    }
}

Back to concurrent programming, inside our main.go file we will instantiate our walletService and walletHandler, create a gin server and instantiate it inside a goroutine:

func main() {

    wService := wallet.NewService()
    wHandler := handler.NewWalletHandler(wService)

    r := gin.New()
    r.Use(gin.Recovery(), gin.Logger())

    r.GET("/", func(c *gin.Context) {
        c.JSON(http.StatusOK, gin.H{
            "message": "Everything is okay here",
        })
    })

    api := r.Group("/api/v1")
    {
        ethNet := api.Group("/eth/wallets")
        {
            ethNet.GET(":address", wHandler.GetWalletByAddress())
            ethNet.GET(":address/transactions", wHandler.GetTransactionsByAddress())
        }
    }

    // Start the Gin server in a goroutine
    go func() {
        if err := r.Run(":8080"); err != nil {
            log.Println("ERROR IN GONIC: ", err.Error())
        }
    }()

    // Contains filtered fields or functions
}

We need start our gin server inside a goroutine to our API and our wallet watcher service run simultaneous, so finally, we will start inside another goroutine our watcher service:

func main() {
    // Filtered fields or functions

    // Start our watcher
    go watcher.StartWatcherService()

    // Wait for the server and the watcher service to finish
    select {}
}

We finalize the implementation of our API and watcher service, as we want the watcher related goroutines to keep running while our application ins running the select{} will wait for their completion, when we created our goroutine for each one of the wallets we include a 'for' with no exit clause, it will be responsible for preventing our goroutines from being completed after their first execution:

//watcher/service.go

func StartWatcherService() {
    // Contains filtered fiels or functions

        go func(wallet domain.Wallet) {
            for {
                // Get the balance of the address
                balance, err := ethClient.BalanceAt(context.Background(), common.HexToAddress(wallet.Address), nil)

                // Contains filtered fields or functions

                time.Sleep(300 * time.Millisecond) // Wait for a while before checking for the next block

                // Contains filtered fields or functions
                }
        }
        // Waiting for all routines to finish - will not cause for is running in an infinity loop.
        wg.Wait()

        // Contains filtered fields or functions
}

Conclusion

The concurrent programming is a powerful tool in the software development, in this article/tutorial step by step we saw how implement it and extract their best in a Ethereum wallet watcher, exchange the 20 wallets for thousands, or the thousands of wallets for thousands of messaging queues or data streams, the go scheduler will take care and use efficiently the available resources. In our example, the 20 goroutines running simultaneous every 300ms consumed 63MB from available memory and the CPU utilization was 6%, this service was running inside a shared instance from 256Mb from Fly.io free tier:

I hope that this article has been useful and has helped you to better understand concurrent programming and goroutines in Go.

Concurrency and Goroutines in Go: Harnessing the power of concurrent programming

Ronilson Alves — Thu, 08 Jun 2023 04:00:00 +0000

In today's fast-paced world, the ability to efficiently perform multiple tasks concurrently is crucial for building robust and high-performance software systems. Go, with its built-in concurrency primitives, provides developers with powerful tools to harness the potential of concurrent programming. In this post, we will explore the concept of concurrency in Go and delve into the world of goroutines, a key feature that makes concurrent programming in Go incredibly expressive and efficient.

Understanding Concurrency:

Concurrency refers to the ability of a program to execute multiple tasks simultaneously, enabling efficient utilization of the available resources. Go embraces concurrency through its lightweight thread-like entities called goroutines. Unlike traditional threads, goroutines are incredibly lightweight and can be created in large numbers without any significant performance overhead. This makes it possible to create thousands of goroutines to perform multiple tasks concurrently without any significant performance degradation.

Goroutines:

Goroutines are functions or methods that can be executed concurrently alongside other goroutines. They can be thought of as independently running units of execution that communicate through channels, a powerful communication mechanism in Go. To create a goroutine, we simply need to prefix the function or method call with the keyword go.
The Go runtime scheduler takes care of managing the execution and scheduling of goroutines across available processor cores.

Let's look at a simple example to understand how goroutines work:

package main

import (
    "fmt"
    "time"
)

func sayHello() {
    for i := 0; i < 5; i++ {
        fmt.Println("Hello")
        time.Sleep(100 * time.Millisecond)
    }
}

func sayWorld() {
    for i := 0; i < 5; i++ {
        fmt.Println("World")
        time.Sleep(100 * time.Millisecond)
    }
}

func main() {
    go sayHello()
    go sayWorld()

    // Wait for goroutines to finish
    time.Sleep(1 * time.Second)
}

In the example above, we have two functions, sayHello and sayWorld, which are executed concurrently as goroutines. The sayHello function prints "Hello" five times, while the sayWorld function prints "World" five times. By executing them concurrently, we can observe interleaved output of "Hello" and "World" in the console.

Syncronization and Communication:

While goroutines enable concurrent execution, proper synchronization and communication between them are essential to avoid data races and ensure consistent results. Go provides channels, a mechanism for safely passing data and synchronizing goroutines. Channels can be used to send and receive values between goroutines, allowing them to communicate and coordinate their actions.

package main

import (
    "fmt"
    "time"
)

func printMessage(message string, ch chan bool) {
    for i := 0; i < 5; i++ {
        fmt.Println(message)
        time.Sleep(100 * time.Millisecond)
    }
    ch <- true // Notify that the goroutine has completed
}

func main() {
    ch := make(chan bool) // Create a channel for synchronization

    go printMessage("Hello", ch)
    go printMessage("World", ch)

    // Wait for both goroutines to finish
    <-ch
    <-ch
}

In this example, we have the printMessage function that takes a message string and a channel ch as parameters. Within the function, the message is printed five times with a small delay between each print statement. After completing its execution, it sends a true value into the channel to signal that it has finished.

In the main function, we create a channel ch using the make function. Then, we launch two goroutines using the go keyword, each executing the printMessage function with different messages ("Hello" and "World") and sharing the same channel ch.

To ensure synchronization and wait for the goroutines to complete, we use the channel operations <-ch. The <-ch operation blocks until a value is received from the channel. By performing this operation twice, we ensure that both goroutines have completed before the program exits.

By using the channel ch, the goroutines communicate and coordinate their actions. They take turns printing their respective messages, resulting in an interleaved output of "Hello" and "World" in the console.

This example demonstrates how channels can be used to synchronize goroutines, allowing them to communicate and coordinate their activities, thereby avoiding data races and ensuring consistent results in concurrent programs.

Conclusion:

Concurrency and goroutines are powerful features of Go that enable efficient and expressive concurrent programming. By leveraging goroutines and channels, developers can build highly performant and scalable systems that make the most of modern hardware capabilities. Understanding the principles of concurrency and using goroutines effectively empowers developers to write concurrent code that is not only efficient but also easier to reason about and maintain.

So, if you're looking to embrace the power of concurrency in your Go programs, give goroutines a try, and unlock a world of highly concurrent and efficient software development. Happy coding!

Microservices: GoLang in a Spring Cloud architecture

Ronilson Alves — Thu, 09 Feb 2023 19:00:00 +0000

When we talk about microservices we think about various little services working together, each one independent that can be implemented in separated. Build a microservice in Go probably is not your first option when we are working in a Spring Cloud architecture where the most participating applications are based in Java. But one of the strongest characteristic of microservice architecture is that you could build different services in different languages, where you could use a more performative language to solve a specific problem.

Bellow we'll see how to implement a microservice built in GoLang into a Spring Cloud architecture, so we will use service discovery, an API Gateway and securize the application with OAuth using a Keycloak server as provider.

The problem

You developped an API for an internal system for a dental clinic, with this API the admin employees can register, read, update and delete data from patients, dentists and
schedulings. The clinic already has a system to manage invoices for their patients - also internal, however there's no integration and comunication between the schedulling
service and invoicing service API.
We need make this integration and comunication between them in Spring Cloud environmnent.

To Do List

Register scheduling service into Eureka Server.
Calls the invoice service each time that a new scheduling has been made or occur an update.
Implement auth througth the API Gateway and authenticate each request, including from invoice service for scheduling service.

Getting started

To follow this article, please clone my GitHub repository with the start code branch (I'm assuming that you already know well Java with Spring Framework and GoLang).
This branch contains scheduling-service (GoLang), invoice-service (Java), Spring Eureka Server and Spring Cloud Gateway

git clone -b starter https://github.com/ronilsonalves/GoLang-in-a-spring-cloud-architecture.git
cd scheduling-service
go mod download

Registering and deregistering our microservice in Spring Eureka

To implement service discovery in our GoLang microservice we will use GoKit, a toolkit for microservices that provides support to auth, log, service discovery, tracing and more.
For this starter code the mod already installed, you can skip this step

go get github.com/go-kit/kit

We need to buid a fargo instance containing all information about our Go Lang microservice before we can register it on Eureka

func buildFargoInstanceBody(appName, status string) *fargo.Instance {
    ipAddr, err := externalIP()
    if err != nil {
        fmt.Println(err)
    }
    port, err := strconv.Atoi(os.Getenv("PORT"))
    if err != nil {
        fmt.Println(err)
        port = 9000
    }
    return &fargo.Instance{
        InstanceId:        ipAddr + ":" + appName + ":" + os.Getenv("PORT"),
        HostName:          "localhost",
        App:               strings.ToUpper(appName),
        IPAddr:            ipAddr,
        VipAddress:        appName,
        SecureVipAddress:  appName,
        Status:            fargo.StatusType(status),
        Overriddenstatus:  "UNKNOWN",
        Port:              port,
        PortEnabled:       true,
        SecurePort:        8443,
        SecurePortEnabled: false,
        HomePageUrl:       "http://localhost:" + os.Getenv("PORT") + os.Getenv("BASE_PATH"),
        StatusPageUrl:     "http://localhost:" + os.Getenv("PORT") + "/status",
        HealthCheckUrl:    "http://localhost:" + os.Getenv("PORT") + "/health",
        CountryId:         0,
        DataCenterInfo: fargo.DataCenterInfo{
            Name: "MyOwn", Class: "com.netflix.appinfo.InstanceInfo$DefaultDataCenterInfo",
        },
        LeaseInfo: fargo.LeaseInfo{},
        Metadata:  fargo.InstanceMetadata{},
        UniqueID:  nil,
    }
}

// BuildFargoInstance build a Fargo Instance and return eureka.Registrar
func BuildFargoInstance() eureka.Registrar {
    eurekaAddr := os.Getenv("EUREKA_SERVER_URL")
    if eurekaAddr == "" {
        fmt.Println("EUREKA_SERVER_URL is not set")
    }

    logger := log.NewLogfmtLogger(os.Stderr)
    logger = log.With(logger, "ts", log.DefaultTimestamp)

    var fargoConfig fargo.Config
    fargoConfig.Eureka.ServiceUrls = []string{eurekaAddr}
    fargoConfig.Eureka.PollIntervalSeconds = 1

    fargoConnection := fargo.NewConnFromConfig(fargoConfig)
    fInstance := buildFargoInstanceBody("scheduling-service", "UP")
    return *eureka.NewRegistrar(&fargoConnection, fInstance, log.With(logger, "component", "registrar"))
}

With eureka.Registrar returned we can use it to register and deregister our microservice, for that at main function we need only atribute the
return of BuildFargoInstance() to a var and call Register() method.

eurekaRegister := sd.BuildFargoInstance()
eurekaRegister.Register()

At this point we've registered successfully our GoLang microservice in Eureka Server, the module provided by GoKit automatically sends heartbeats to
our Eureka Server instance while our microservice is running, but to update and deregister our microservice in case of his instance is terminated or killed
we need call Deregister() method, for that we will build a channel to monitor the application status in our main function.

c := make(chan os.Signal)
signal.Notify(c, os.Interrupt)
// contains filtered fields or functions in main function
go func() {
    select {
    case signal := <-c:
        _ = signal
        time.Sleep(4 * time.Second)
        eurekaRegister.Deregister()
        os.Exit(1)
    }
}()

We've successfully integrated our GoLang microservice into Spring Eureka. The first item of our to do list is done.

Calling invoice service through a RabbitMQ queue

To call our invoicing service each time an event occur in scheduling service we will use a RabbitMQ queue, each event
will send a message to our queue and in invoice service we will implement a listener for this queue and consume this message.

First all, in our scheduling service we need to implement a publisher, for that we will use a go rabbitmq client. Again,
this starter code already has a module, so the following step is not necessary unless you pretend to use another client or
building a new project.

go get github.com/hadihammurabi/go-rabbitmq

So, we need to connect to our RabbitMQ instance and create a channel to publish our messages.

// ConnectRabbitMQ connect and setup RabbitMQ channel and queue
func ConnectRabbitMQ(urlConn, name string) (gorabbitmq.MQ, error) {
    mq, err := gorabbitmq.New(urlConn)
    failOnError(err, "Failed to create a MQ")

    err = mq.Exchange().
        WithName(name).
        WithType(exchange.TypeDirect).
        Declare()
    failOnError(err, "Failed to create a channel")

    q, err := mq.Queue().
        WithName(name).
        Declare()
    failOnError(err, "Failed to create a queue")

    err = q.Binding().
        WithExchange(name).
        Bind()
    failOnError(err, "Failed to bind queue")

    return *mq, nil
}

In our service we will publish a message to our queue each time a new event is created, for that we need to implement a method to publish a message
to our queue.

// PublishMessage - send a msg to RabbitMQ queue when an appointment is made or updated
func PublishMessage(a domain.AppointmentDTO) {
    mq, err := ConnectRabbitMQ(os.Getenv("RABBIT_MQ_URL_CONN"), "appointment-service")
    log.Println(err)
    body, _ := json.Marshal(a)
    err = mq.Publish(&gorabbitmq.MQConfigPublish{
        RoutingKey: mq.Queue().Name,
        Message: amqpi.Publishing{
            ContentType: "application/json",
            Body:        body,
        },
    })
    defer mq.Close()
}

Now we need to call this method each time a new event is created or updated, for that we need call it in our Create() and Update() methods in our service.

func (s *service) Create(a domain.Appointment)(domain.AppointmentDTO, error) {
    // contains filtered fields or functions
    if ok {
        amqp.PublishMessage(apSaved)
        return apSaved, nil
    }
}

func (s *service) Update(id int, a domain.Appointment)(domain.AppointmentDTO, error) {
    // contains filtered fields or functions
    amqp.PublishMessage(response)
    return response, nil
}

We already have our publisher implemented, now we need to implement our listener/consumer in our invoice service. As business rule, we need create an invoice
each time a new scheduling is made and it is updated we need update the invoice.

First we will configure our RabbitMQ connection and create a channel to consume our messages.

@Configuration
public class RabbitMQSenderConfig {
    @Value("${queue.appointment-service.name}")
    private String appointmentServiceQueue;

    @Bean
    public Queue appointmentQueue() {
        return new Queue(this.appointmentServiceQueue,false);
    }
}

Also we need setup a message converter to convert our message to a Java object. As in this project we are using Brazilian Datetime format("dd/MM/yyyy HH:mm"), we need to
configure a customObjectMapper to avoid an exception while trying to convert Datetime fields. For that we will create a class.

@RequiredArgsConstructor
public class CustomLocalDateTimeObjectMapper {
    private final ObjectMapper customObjectMapper;

    public ObjectMapper getCustomLocalDateTimeObjectMapper() {
        JavaTimeModule module = new JavaTimeModule();
        this.customObjectMapper.setVisibility(PropertyAccessor.ALL, JsonAutoDetect.Visibility.ANY);
        this.customObjectMapper.disable(SerializationFeature.WRITE_DATES_AS_TIMESTAMPS);
        LocalDateTimeDeserializer localDateTimeDeserializer =
                new LocalDateTimeDeserializer(DateTimeFormatter.ofPattern("dd/MM/yyyy HH:mm"));
        module.addDeserializer(LocalDateTime.class, localDateTimeDeserializer);
        this.customObjectMapper.registerModule(module);
        return this.customObjectMapper;
    }
}

And now we need to configure our RabbitMQ template to use our customObjectMapper.

@Configuration
public class RabbitTemplateConfig {

    @Bean
    public Jackson2JsonMessageConverter producerJackson2MessageConverter() {
        CustomLocalDateTimeObjectMapper customLocalDateTimeObjectMapper =
                new CustomLocalDateTimeObjectMapper(new ObjectMapper());
        return new Jackson2JsonMessageConverter(customLocalDateTimeObjectMapper.getCustomLocalDateTimeObjectMapper());
    }

    @Bean
    public RabbitTemplate rabbitTemplate(ConnectionFactory connectionFactory) {
        RabbitTemplate rabbitTemplate = new RabbitTemplate(connectionFactory);
        rabbitTemplate.setMessageConverter(producerJackson2MessageConverter());
        return rabbitTemplate;
    }
}

Now we need to create a listener to consume our messages.

@RequiredArgsConstructor
@Component
public class AppointmentListener {
    private final Logger logger = Logger.getLogger(AppointmentListener.class.getName());
    private final InvoiceServiceImpl invoiceService;

    @RabbitListener(queues = {"${queue.appointment-service.name}"})
    public void receiveMessage(Appointment appointment) {
        logger.log(Level.INFO,"Message received from RabbitMQ: "+appointment);
        List<Invoice> invoices =
                invoiceService.listAll().stream().filter(invoice -> Objects.equals(invoice.getAppointmentId(), appointment.id())).toList();
        if (invoices.isEmpty()) {
            invoiceService.save(appointment);
        } else {
            invoiceService.update(invoices.get(0),appointment);
        }
    }
}

In our listener we verify if there is an invoice for the appointment, if there is not we create a new invoice, if there is we update the invoice.
And that point we have our invoice service listening to our appointment service and creating or updating invoices. Second item of our todo list is done.
But comunication not yet, we will finish after update our authentication flow.

Securizing our scheduling service with Keycloak as Identity Provider.

This is the third item of our todo list, we need to secure our scheduling service with Keycloak as Identity Provider. As we are using Spring Gateway and Spring Security
we can setup a token relay and instead of using a simple token we can use the JWT token provided by Keycloak and validate it in our scheduling service if the user has the
authorized role.

The Keycloak realm configuration is available in the GitHub repository.

First we need to configure our Spring Gateway to relay the token to our scheduling service. We can setup it in our application.yml file. Note: the application.yml file already
contains the configuration for token relay, Keycloak, so you can skip this step if you are using the same file from the GitHub repository.

## contains filtered fields
  cloud:
    gateway:
      routes:
        - id: invoice-service
          uri: lb://invoice-service
          predicates:
            - Path=/invoices/**
        - id: scheduling-service
          uri: lb://scheduling-service
          predicates:
            - Path=/api/v1/**
      default-filters:
        - TokenRelay
        - LogFilter
## contains filtered fields

In our SecurityConfig.java we have configured that any request must be authenticated, if no, the user will be redirected to Keycloak login page.

@Configuration
public class SecurityConfig {

    @Bean
    public SecurityWebFilterChain springSecurityFilterChain(ServerHttpSecurity httpSecurity) {
        httpSecurity
                .authorizeExchange()
                .pathMatchers("/actuator/**").permitAll()
                .anyExchange().authenticated()
                .and()
                .oauth2Login()
                .and().csrf().disable();
        return httpSecurity.build();
    }
}

In our scheduling service we need to change the authentication to use the JWT token provided by Keycloak and relayed by our gateway. We must create a middleware to validate the token received.

type Claims struct {
    RealmAccess roles  `json:"realm_access,omitempty"`
    JTI         string `json:"jti,omitempty"`
}

type roles struct {
    Roles []string `json:"roles,omitempty"`
}

var RealmConfigURL = os.Getenv("REALM_CONFIG_URL")
var clientID = os.Getenv("CLIENT_ID")
var authorizedRole = "ADMIN"

func IsAuthorizedJWT() gin.HandlerFunc {
    return func(c *gin.Context) {
        rawAccessToken := strings.Replace(c.GetHeader("Authorization"), "Bearer", "", 1)

        trans := &http.Transport{
            TLSClientConfig: &tls.Config{
                InsecureSkipVerify: true,
            },
        }
        client := &http.Client{
            Timeout:   time.Duration(6000) * time.Second,
            Transport: trans,
        }

        ctx := oidc.ClientContext(context.Background(), client)
        provider, err := oidc.NewProvider(ctx, RealmConfigURL)
        if err != nil {
            authorizationFailed("an authorization error occurred while getting the provider: "+err.Error(), c)
            return
        }

        oidcConfig := &oidc.Config{
            ClientID: clientID,
        }

        verifier := provider.Verifier(oidcConfig)
        idToken, err := verifier.Verify(ctx, rawAccessToken)
        if err != nil {
            authorizationFailed("an authorization error occurred while verifying the token: "+err.Error(), c)
            return
        }

        var IDTokenClaims Claims
        if err := idToken.Claims(&IDTokenClaims); err != nil {
            authorizationFailed("An error occurred while extracting claims: "+err.Error(), c)
            return
        }

        userAccessRoles := IDTokenClaims.RealmAccess.Roles
        for _, userRole := range userAccessRoles {
            if userRole == authorizedRole {
                c.Next()
                return
            }
        }

        authorizationFailed("The user has no permission to access this API", c)
    }

}

func authorizationFailed(message string, c *gin.Context) {
    web.BadResponse(c, http.StatusUnauthorized, "ERROR", message)
    return
}

So in our main.go we need to updated the authentication middleware to use the JWT token in our router.

func main() {
    // contains filtered fields or functions
    r.Use(IsAuthorizedJWT())
    // contains filtered fields or functions
}

Now, the access to our scheduling service is secured througth our API Gateway. The third item of our todo list is almost done.

If an user try to create an invoice manually, he must provide an appointment ID and a price in request body, with ID provided our invoice service will request the scheduling service
to get the appointment data and validate if the appointment exists and after all create the invoice, to make this request we will use a Feign client and it must be authenticated.

We need handle the client authentication.

@Configuration
@RequiredArgsConstructor
public class FeignConfiguration {

    private static final String KEYCLOAK_REGISTRATION_ID = "keycloak-registration";

    private final OAuth2AuthorizedClientService clientService;
    private final ClientRegistrationRepository registrationRepository;

    @Bean
    public RequestInterceptor requestInterceptor() {
        ClientRegistration clientRegistration = registrationRepository.findByRegistrationId(KEYCLOAK_REGISTRATION_ID);
        OAAuth2ClientCredentialsFeignManager credentialsFeignMananger =
                new OAAuth2ClientCredentialsFeignManager(authorizedClientManager(),clientRegistration);
        return requestInterceptor -> {
            requestInterceptor.header("Authorization", "Bearer " + credentialsFeignMananger.getAccessToken());
        };
    }

    @Bean
    public OAuth2AuthorizedClientManager authorizedClientManager () {
        OAuth2AuthorizedClientProvider authorizedClientProvider = OAuth2AuthorizedClientProviderBuilder
                .builder()
                .clientCredentials()
                .build();

        AuthorizedClientServiceOAuth2AuthorizedClientManager authorizedClientManager =
                new AuthorizedClientServiceOAuth2AuthorizedClientManager(registrationRepository, clientService);

        authorizedClientManager.setAuthorizedClientProvider(authorizedClientProvider);

        return authorizedClientManager;
    }
}

Since our invoice service may request data from appointments and consequently clients and dentists related to theses appointments
using feign clients. As all endpoints of our scheduling API require authentication, we need to setup a role to the user from our keycloak invoice service client.

NOTE: In the keycloak realm configuration provided at the GitHub repository, the configuration already has been made, you can skip this step.
The keycloak version used for this tutorial is the 20.0.2 running in a docker container

Step 1: Access the admin console from your keycloak server

Step 2: Access the backend-services client. In Settings, go to Capability config and check the Service accounts roles option

Step 3: Access the Realm roles. Create a new role called ADMIN (the same that we are mapping in our microservices)

Step 4: Access the Service account roles tab from backend-services client and assign the ADMIN role to it

Don't forget to create a user in your keycloak realm and assign the ADMIN role to it.

And done, we have successfully secured our scheduling service and integrated it into Spring Cloud architecture.

I hope that this tutorial was useful to you and if you have any questions or suggestions, please let me know. Thanks for reading.