DEV Community: Wagner Manganelli (aka manga)

Crossplane + ArgoCD: A Perfect Match for Kubernetes Automation

Wagner Manganelli (aka manga) — Mon, 23 Sep 2024 21:54:19 +0000

Introduction

Automation is key to managing complex infrastructures and applications efficiently. Two powerful tools that help achieve this are Crossplane and ArgoCD. While Crossplane enables you to manage cloud infrastructure declaratively using Kubernetes, ArgoCD brings GitOps principles to automate the deployment and management of applications.

In this step-by-step guide, I’ll show you how to set up and integrate these two tools on a local Kubernetes cluster using Kind. Whether you're experimenting locally or preparing for a production setup, this tutorial will walk you through the process of provisioning infrastructure and managing your applications with ease. Let’s get started!

If you're new to GitOps or ArgoCD, I highly recommend checking out my previous post: GitOps + ArgoCD: A Perfect Match for Kubernetes Continuous Delivery for an introduction to these concepts.

Hands-on 🚀

Prerequisites

Clone the Hands-on GitHub Repository



git clone git@github.com:mangadev-dev-to/crossplane-argocd.git

Folder Structure



├── crossplane
├── gitops
├── kind
└── rendered-resources

crossplane: Contains Crossplane configuration, from installation to resources.
gitops: The source of truth for all application manifests (typically stored in a separate repository).
kind: Contains local cluster configuration.
rendered-resources: Contains the rendered Crossplane resources manifests.

Create a Kind Cluster



kind create cluster --config=kind/kind.yaml --name=crossplane-argocd

Install Crossplane: Install Crossplane on your Kind cluster.



# Ensure you're in the Kind cluster. This command should return 'kind-crossplane-argocd' context.
kubectl config current-context

# Add the Crossplane repository
helm repo add crossplane-stable https://charts.crossplane.io/stable

# Update the local Helm chart cache
helm repo update

# Install the Crossplane Helm chart 
helm install crossplane --namespace crossplane-argocd --create-namespace crossplane-stable/crossplane

# Grant cluster-admin role to Crossplane service account (use with caution in production)
kubectl apply -f crossplane/rbac/clusterrole-admin-binding-crossplane.yaml

Install Crossplane Providers: Install Kubernetes and Helm Providers.



kubectl apply -f crossplane/provider/k8s-provider.yaml 
kubectl apply -f crossplane/provider/k8s-provider-config.yaml
kubectl apply -f crossplane/provider/helm-provider.yaml
kubectl apply -f crossplane/provider/helm-provider-config.yaml

# Grant cluster-admin role to Providers service accounts (use with caution, it's not recommended for production environments)
kubectl apply -f crossplane/rbac/clusterrole-admin-binding-kubernetes-provider.yaml
kubectl apply -f crossplane/rbac/clusterrole-admin-binding-helm-provider.yaml

Provision Crossplane ArgoCD Instance: Create an instance of ArgoCD by applying the Kubernetes Object Resource along with Helm.



# Create argocd namespace and context
kubectl create namespace argocd
kubectl config set-context kind-ns-argocd --namespace argocd --cluster kind-crossplane-argocd --user kind-crossplane-argocd

# Set current context for argocd namespace
kubectl config use-context kind-ns-argocd

# Apply k8s object with ArgoCD Instance Helm
kubectl apply -f crossplane/argocd/argocd-instance.yaml

# Validate if it is up and running (it may take some seconds to get ready)
kubectl get releases.helm.crossplane.io

# Grant cluster-admin role to the ArgoCD service account (use with caution in production)
kubectl apply -f crossplane/argocd/rbac/argocd-svc-account-clusterrole-admin-binding.yaml

# Get the admin password via kubectl
kubectl get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d

# Access the ArgoCD UI (http://localhost:8080) using 'admin' as the username and the copied password
kubectl port-forward service/argocd-instance-server 8080:443

Install Crossplane ArgoCD Provider: Install and configure ArgoCD Provider.



# Create a new ArgoCD user for the provider
kubectl patch configmap/argocd-cm --type merge -p '{"data":{"accounts.provider-argocd":"apiKey"}}'
kubectl patch configmap/argocd-rbac-cm --type merge -p '{"data":{"policy.csv":"g, provider-argocd, role:admin"}}'

# Create an API Token
ARGOCD_ADMIN_SECRET=$(kubectl get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d)

# Port forward the ArgoCD API to the host
kubectl port-forward service/argocd-instance-server 8080:443

# Create a session JWT for the admin user at the ArgoCD API
ARGOCD_ADMIN_TOKEN=$(curl -s -X POST -k -H "Content-Type: application/json" --data '{"username":"admin","password":"'$ARGOCD_ADMIN_SECRET'"}' https://localhost:8080/api/v1/session | jq -r .token)

# Create an API token without expiration for provider-argocd
ARGOCD_PROVIDER_USER="provider-argocd"
ARGOCD_TOKEN=$(curl -s -X POST -k -H "Authorization: Bearer $ARGOCD_ADMIN_TOKEN" -H "Content-Type: application/json" https://localhost:8080/api/v1/account/$ARGOCD_PROVIDER_USER/token | jq -r .token)

# Create a Kubernetes secret from the JWT so provider-argocd can connect to ArgoCD
kubectl create secret generic argocd-credentials -n crossplane-argocd --from-literal=authToken="$ARGOCD_TOKEN"

# Install ArgoCD Provider
kubectl apply -f crossplane/provider/argocd-provider.yaml

# Configure a ProviderConfig with serverAddr pointing to the ArgoCD Instance
kubectl apply -f crossplane/provider/argocd-provider-config.yaml

# Validate if ArgoCD Provider is healthy
kubectl get providers.pkg.crossplane.io crossplane-provider-argocd

# Grant cluster-admin role to ArgoCD Provider service account (use with caution in production)
kubectl apply -f crossplane/rbac/clusterrole-admin-binding-argocd-provider.yaml

Prepare ArgoCD Resources: Set up ArgoCD Projects and Applications



# Render Helm Charts
./resource-helm-template-generator.sh car
./resource-helm-template-generator.sh truck
./resource-helm-template-generator.sh train

# Validate if the rendered Helm Charts are present in the 'rendered-resources' folder

Create ArgoCD Resources: Apply the rendered resources



# Apply the rendered resources
kubectl apply -f rendered-resources/

# Validate the Crossplane objects
kubectl get objects.kubernetes.crossplane.io
kubectl get projects.projects.argocd.crossplane.io
kubectl get applications.applications.argocd.crossplane.io

# Also, validate in the ArgoCD UI (http://localhost:8080) that the applications are up and running

Time to BREAK THINGS: Let Crossplane work its reconciliation magic 🪄

Delete the car-app-1 using the ArgoCD UI.
Crossplane shall reconcile and recreate it (it may take some seconds)
After some seconds the car-app-1 will be recreated and showed in the UI again.



# Check the Kubernetes events for more details
kubectl events

8m54s:
- The application car-app-1 transitioned from "Progressing" to "Healthy" status, indicating it reached a stable state.
90s:
- The application car-app-1 transitioned from "Progressing" to "Missing," suggesting that the resource went missing or was deleted.
- An admin deleted the application car-app-1.
- The application's health status changed from "Healthy" to "Progressing," showing a new change or re-deployment began.
- The sync status changed from "Synced" to "OutOfSync," indicating the application's actual state no longer matched the desired state defined in Git.
65s:
- The sync status of car-app-1 was updated to "OutOfSync," confirming it was still not in sync.
- The health status of car-app-1 changed to "Missing," meaning the resource was not present.
- An automated sync operation was initiated to commit 29c47ab08e8e14c95a1124b2aca7d3c47a50f6ee.
- The provider provider-argocd created the application car-app-1.
64s:
- The application's sync status changed from "OutOfSync" to "Synced," meaning it matched the desired state from Git again.
- The health status moved from "Missing" to "Progressing," indicating it was being set up or restored.
- The sync operation to commit 29c47ab08e8e14c95a1124b2aca7d3c47a50f6ee completed successfully.
61s:
- The application car-app-1 transitioned from "Progressing" to "Healthy," meaning it reached a stable and desired state once more.

Final Thoughts

In the previous post GitOps + ArgoCD: A Perfect Match for Kubernetes Continuous Delivery, we explored how to automate application deployments using GitOps with ArgoCD. Now, we've taken it a step further. As demonstrated, ArgoCD applications are actively monitored and managed by Crossplane, ensuring that any accidental deletions by a developer or unexpected incidents are swiftly corrected. Crossplane automatically reconciles the state and re-establishes the desired configuration, providing enhanced robustness, resilience, and reliability for your infrastructure.

By integrating Crossplane with ArgoCD, you've created a powerful, self-healing environment that maintains your desired state, reducing downtime, mitigating risks, and enhancing operational efficiency.

That's it! Happy coding! 🎉🎉🎉

GitOps + ArgoCD: A Perfect Match for Kubernetes Continuous Delivery

Wagner Manganelli (aka manga) — Sat, 21 Sep 2024 20:21:23 +0000

Introduction

Deploying and managing Kubernetes applications can be complex, but GitOps simplifies it by treating your Git repository as the single source of truth for deployments. ArgoCD streamlines this process further by automatically syncing your Kubernetes clusters with your Git repository.

In this guide, we’ll walk you through setting up GitOps with ArgoCD, making your Kubernetes deployments fast, consistent, and automated. If you're aiming to simplify your workflow, stay tuned—this tutorial will get you up and running in no time.

To dive deeper into GitOps, check out this detailed blog post by @sujal_dua:
GitOps Adoption: A Simple Guide to Modern Cloud Management.

Hands-on 🚀

Prerequisites

Docker Desktop
kubectl
Helm
Kind
VS Code (or any code editor)

Clone the Hands-on GitHub Repository

git clone git@github.com:mangadev-dev-to/gitops-argocd.git

Folder Structure

├── argocd
├── gitops
├── kind
└── rendered-resources

gitops: The source of truth for all application manifests (typically stored in a separate repository).
kind: Contains local cluster configuration.
argocd: Contains ArgoCD configuration, from installation to resources.
rendered-resources: Contains the rendered ArgoCD manifests.

Create a Kind Cluster

kind create cluster --config=kind/kind.yaml --name=gitops-argocd

Install ArgoCD: Install ArgoCD on your Kind cluster

# Ensure you're in the Kind cluster. This command should return 'kind-gitops-argocd' context.
kubectl config current-context

# Add the ArgoCD Helm repository
helm repo add argo https://argoproj.github.io/argo-helm

# Update the local Helm chart cache
helm repo update

# Install the ArgoCD Helm chart
helm install argocd --namespace argocd --create-namespace argo/argo-cd

# Create context for the ArgoCD namespace
kubectl config set-context kind-ns-argocd --namespace argocd --cluster kind-gitops-argocd --user kind-gitops-argocd

# Set the current context for the argocd namespace
kubectl config use-context kind-ns-argocd

# Grant cluster-admin role to the ArgoCD service account (use with caution in production)
kubectl apply -f argocd/rbac/argocd-svc-account-clusterrole-admin-binding.yaml

# Get the admin password via kubectl
kubectl get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d

# Access the ArgoCD UI (http://localhost:8080) using 'admin' as the username and the copied password
kubectl port-forward service/argocd-server 8080:443

Prepare ArgoCD Resources: Set up ArgoCD Projects and Applications

# Render Helm Charts
./resource-helm-template-generator.sh car
./resource-helm-template-generator.sh truck
./resource-helm-template-generator.sh train

# Validate if the rendered Helm Charts are present in the 'rendered-resources' folder

Create ArgoCD Resources: Apply the rendered resources

# Create the rendered Helm Charts
kubectl apply -f rendered-resources/

# Validate in the ArgoCD UI (http://localhost:8080) that the applications are up and running

Update Application: Make changes to the application and let ArgoCD handle reconciliation

Open the manifest gitops/car/app-1/deployment.yaml and change the replicas value to 2 on line 38.
Save and push the changes to GitHub repository.
Wait 3 minutes or manually sync the car-app-1 in ArgoCD.
Verify that the car application now has two pods running.

Before Changes:

After Changes:

Final Thoughts

For simplicity, we kept the GitOps application manifests in the same repository and made changes manually, without using automation tools like GitHub Actions, Azure DevOps, or Jenkins.
ArgoCD offers an intuitive UI, enabling development teams to manage their applications effortlessly.
With just a few steps, you've enhanced application management, improving robustness, observability, and maintainability using a declarative approach.

That's it! Happy coding! 🎉🎉🎉

No More Passwords! OIDC Terraform Module Makes GCP-GitHub Authentication a Breeze

Wagner Manganelli (aka manga) — Wed, 01 May 2024 17:48:45 +0000

Intro

Hello there! Welcome to our guide on automating OpenID Connect (OIDC) using Terraform with Google Cloud Platform (GCP) to grant access to GitHub Actions.

Overview

In this post, we'll explore the seamless integration of OIDC, enabling GitHub Actions workflows to access GCP resources without the need to store long-lived GCP credentials as GitHub secrets.

Prerequisites

Before we begin, ensure you have the following prerequisites:

Terraform CLI installed
gcloud CLI installed
Access to a Google Cloud Platform (GCP) project
Access to a GitHub repository

Next Steps

Let's jump into the configuration process to make this integration work seamlessly.

Setting Up Terraform:

Create a Project Folder: Start by creating a folder for your Terraform configuration and navigate into it:

mkdir terraform-oidc
cd terraform-oidc

Set Terraform Variables: Create a file called variables.tf and past the following configuration into it:

variable "project" {
  default = "dev-to-oidc" // replace with your project id
}

variable "credentials_file" {
  default = "~/.config/gcloud/application_default_credentials.json" // replace with your credentials path
}

variable "region" {
  default = "us-central1" // replace with your region
}

variable "zone" {
  default = "us-central1-f" // replace with your zone
}

variable "gh_repo" {
  default = "manganellidev/dev-to-oidc-gcp-tf" // replace with your organization/repository
}

Set Terraform Configuration: Create a file called main.tf and past the following Terraform configuration into it:

terraform {
  required_providers {
    google = {
      source  = "hashicorp/google"
      version = "5.27.0"
    }
  }
}

provider "google" {
  credentials = file(var.credentials_file)
  project     = var.project
  region      = var.region
  zone        = var.zone
}

// enable iam credentails api
resource "google_project_service" "iam_credentials_api" {
  project = var.project
  service = "iamcredentials.googleapis.com"
  disable_on_destroy = false
}

// create service account
resource "google_service_account" "oidc_service_account" {
  project      = var.project
  account_id   = "oidc-service-account"
  display_name = "OIDC Service Account"
  description  = "This service account is used for my application to interact with Google Cloud services."
}

// create and configure oidc resources
module "gh_oidc" {
  source      = "terraform-google-modules/github-actions-runners/google//modules/gh-oidc"
  project_id  = var.project
  pool_id     = "oidc-pool"
  provider_id = "oidc-gh-provider"
  attribute_mapping = {
    "attribute.repository": "assertion.repository",
    "google.subject": "assertion.sub"
  }
  sa_mapping = {
    "oidc-service-account" = {
      sa_name   = "projects/${var.project}/serviceAccounts/${google_service_account.oidc_service_account.email}"
      attribute = "attribute.repository/${var.gh_repo}"
    }
  }
}

output "service_account_email" {
  value = google_service_account.oidc_service_account.email
}

Initialize Terraform:

terraform init

Login to GCP:

gcloud auth login

Set target GCP project:

# replace dev-to-oidc with your project id
gcloud config set project dev-to-oidc

Apply Terraform:

terraform apply

# Review the changes than type yes + enter
# Copy the service account email from the output in the terminal and save it to be used later (e.g oidc-service-account@dev-to-oidc.iam.gserviceaccount.com)

Get Workload Identity Provider:

gcloud iam workload-identity-pools providers list --location="global" --workload-identity-pool="oidc-pool"

# Copy the name value and save it to be used later (e.g projects/123123123123/locations/global/workloadIdentityPools/oidc-pool/providers/oidc-gh-provider)

Setting Up GitHub Actions:

Create Github Workflow:

mkdir .github
mkdir .github/workflows
touch .github/workflows/workflow-test.yml

on:
  workflow_call:

  push:
    branches:
      - "main"

jobs:
  auth-oidc:
    runs-on: ubuntu-latest

    permissions:
      id-token: write
      contents: read

    steps:
      - name: Google Auth
        uses: google-github-actions/auth@v2
        with:
          token_format: access_token
          project_id: dev-to-oidc
          service_account: oidc-service-account@dev-to-oidc.iam.gserviceaccount.com # replace with your service account name
          workload_identity_provider: projects/123123123123/locations/global/workloadIdentityPools/oidc-pool/providers/oidc-gh-provider # replace with your WIF provider name

      - name: "Set up Cloud SDK"
        uses: "google-github-actions/setup-gcloud@v2"

      - name: Use gcloud CLI
        run: |
          gcloud auth list --filter=status:ACTIVE --format="value(account)"

Testing the Integration:

Commit everything and push to Github:
The workflow should start automatically. You can verify my workflow run.

Also, you can clone my Github repository:

git clone https://github.com/mangadev-dev-to/oidc-gcp-tf.git

Conclusion:

With this Terraform module, you can streamline the authentication process between GitHub Actions and Google Cloud Platform, eliminating the need for managing and storing sensitive credentials. Stay tuned for more tips and tricks on optimizing your cloud workflows!

That's it! Happy coding! 🎉🎉🎉

Hands-on Kafka: First Steps

Wagner Manganelli (aka manga) — Mon, 04 Mar 2024 15:21:00 +0000

Embark on a journey into real-time data processing with Kafka! Discover how to set up Kafka clusters, build streaming applications, and unlock new insights for your projects. Whether you're a seasoned developer or a curious enthusiast, this hands-on guide will empower you to harness the full potential of Kafka.

What is Kafka in a nutshell?

Kafka is a distributed streaming platform designed for handling real-time data feeds with high throughput and fault tolerance. It operates through servers and clients communicating via a high-performance TCP network protocol. Kafka can be deployed on various infrastructures including bare-metal hardware, virtual machines, and containers, both on-premise and in cloud environments.

To comprehend Kafka thoroughly, it's essential to understand these key definitions:

Topic: Think of a topic as a category or a channel where messages are organized and published. It's like a virtual inbox or bulletin board where messages related to a specific theme or subject are posted.
Event: Represents a piece of data or a message denoting a specific action or occurrence, which is transmitted to a topic.
Producer: Acts as a sender or publisher, generating and dispatching events to topics.
Consumer: Is like a receiver or a subscriber that reads and processes events from topics. It's responsible for fetching and handling messages from the topics it subscribes to.

How does it work?

Let's illustrate Kafka's functionality using an e-commerce scenario with a distributed system:

Order Service: Responsible for managing product listings and the checkout process.
Inventory Service: Handles updates to product quantities
Notification Service: Sends notifications to users
Logging Service: Tracks activities across services

In this scenario, we define topics, producers, consumers, and the event contract as follows:

Topics

new-order-topic: Receives events related to new orders
inventory-empty-topic: Receives events regarding product inventory consumption
new-logging-action-topic: Receives events related to new actions from other services

Consumers

Inventory Service: Consumes events from new-order-topic
Notification Service: Consumes events from new-order-topic and inventory-empty-topic
Logging Service: Consumes events from new-logging-action-topic

Producers

Order Service: Produces events for new-order-topic and new-logging-action-topic
Inventory Service: Produces events for inventory-empty-topic and new-logging-action-topic
Notification Service: Produces events for new-logging-action-topic

Events Contract per Topic

new-order-topic:

{
  orderId: string,
  customerId: string,
  products: [
    {
      productId: string,
      quantity: number,
      totalPrice: number
    }
  ]
}

// Sample
{
  "orderId": "ORDER-123",
  "customerId": "CUSTOMER-123",
  "products": [
    {
      "productId": "PRODUCT-123",
      "quantity": 3,
      "totalPrice": 80
    },
    {
      "productId": "PRODUCT-321",
      "quantity": 1,
      "totalPrice": 233
    }
  ]
}

inventory-empty-topic:

{
  productId: string,
}

// Sample
{
  "productId": "PRODUCT-123"
}

new-logging-action-topic:

{
  serviceName: string,
  action: string,
}

// Sample
{
  "serviceName": "order-service",
  "action": "new-order",
}

The diagram below illustrates how it works:

That's it, we have all in place to start the hands-on! 🚀🚀🚀

Hands-on

Prerequisites

To follow this guide, ensure you have the following requirements:

Node.js 18+: If you haven't installed Node.js on your machine yet, you can download the installer from the official Node.js website. Once downloaded, launch the installer and follow the installation wizard.
Docker: Docker is required to set up Kafka on your computer. If you haven't installed Docker yet, you can follow the installation guide for your operating system provided in the official Docker documentation.

The following sub-chapters outline the main steps to start using Kafka with Node.js. To avoid confusion and ensure a smooth process, we recommend keeping the codebase of the final application accessible. You can achieve this by cloning the GitHub repository associated with this article:

git clone https://github.com/mangadev-dev-to/hands-on-kafka.git

This will give you easy access to the code examples and configurations used throughout the guide.

Set Up Your Kafka Project

Create a Project Folder: Start by creating a folder for your Node.js application and navigate into it:

mkdir hands-on-kafka
cd hands-on-kafka

Initialize npm Project: Run the following command to bootstrap a new npm project:

npm init -y

Install Dependencies: Install kafkajs to your project's dependencies:

npm install kafkajs

Add Docker Configuration: Copy the docker-compose.yml file from the GitHub repository to the root folder of your project.

Organize Project Structure: Create a src/ folder and within it, create an index.js file. Additionally, create the following subfolders: lib/, db/, and services/.

Create Modules: Inside the services/ folder, create a module for each service. Similarly, create modules for the Kafka client and the in-memory database. Here's how your project directory should be structured:

hands-on-kafka/
├── node_modules/
├── src/
│    └── db
│      └── inventory.js
│    └── lib
│      └── kafka-client.js
│    └── services
│      └── order-service.js
│      └── inventory-service.js
│      └── notification-service.js
│      └── logging-service.js
│      └── kafka-client.js
│    └── index.js
├── docker-compose.yml
├── package-lock.json
├── package.json
└── README.md

Update package.json

Enable ES Modules: Add the type configuration to enable ES Modules:

"type": "module"

Add Start Script: To facilitate running your application, add a start script to the scripts object:

"start": "node src/index.js"

Index

This file serves as the entry point for the application. It imports all services responsible for consuming and publishing events related to various aspects of the e-commerce system. Additionally, it continuously generates new order events every 5 seconds to simulate ongoing activity within the system.

// Import services responsible for consuming and publishing events
import "./services/inventory-service.js";
import "./services/logging-service.js";
import "./services/notification-service.js";

// Import function to create new order events
import createNewOrder from "./services/order-service.js";

// Define initial order details
const orderOne = {
  orderId: "ORDER-123",
  customerId: "CUSTOMER-123",
  products: [
    {
      productId: "PRODUCT-123",
      quantity: 3,
      totalPrice: 80,
    },
    {
      productId: "PRODUCT-321",
      quantity: 1,
      totalPrice: 233,
    },
  ],
};

// Create the initial order and schedule creation of new orders every 5 seconds
(async () => {
  await createNewOrder(orderOne);

  // Schedule creation of new orders recursively every 5 seconds
  setInterval(() => createNewOrder(orderOne), 5000);
})();

Kafka Client Module

This module configures the Kafka client by specifying the required parameters. Ensure you have the following configuration added:

import { Kafka } from "kafkajs";

// Define constants for client ID and Kafka brokers URL
const CLIENT_ID = "kafka-hands-on";
const BROKERS_URL = ["localhost:29092"];

// Define topics used in the application
export const TOPIC_NEW_ORDER = "new-order-topic";
export const TOPIC_INVENTORY_EMPTY = "inventory-empty-topic";
export const TOPIC_NEW_LOGGING_ACTION = "new-logging-action-topic";

// Initialize Kafka client with specified configuration
const kafkaClient = new Kafka({ clientId: CLIENT_ID, brokers: BROKERS_URL });

export default kafkaClient;

Ensure that the CLIENT_ID and BROKERS_URL variables are correctly set based on your Kafka configuration. This module provides a centralized configuration for the Kafka client, making it easier to manage and reuse throughout your application.

Services Module

Order Service

This service module contains a single function responsible for initiating the messaging process by creating a new order event. Consumers will receive this event asynchronously to perform additional processing.

import kafkaClient, {
  TOPIC_NEW_ORDER,
  TOPIC_NEW_LOGGING_ACTION,
} from "../lib/kafka-client.js";

// Initialize Kafka producer
const producer = kafkaClient.producer();

const createNewOrder = async (input) => {
  try {
    // Connect to Kafka broker
    await producer.connect();

    // Create a new event for the 'new-order-topic'
    await producer.send({
      topic: TOPIC_NEW_ORDER,
      messages: [{ value: JSON.stringify(input) }],
    });

    // Create a new event for the 'new-logging-action-topic'
    await producer.send({
      topic: TOPIC_NEW_LOGGING_ACTION,
      messages: [
        {
          value: JSON.stringify({
            serviceName: "order-service",
            action: "new-order",
          }),
        },
      ],
    });
  } catch (err) {
    console.error(err);
  } finally {
    await producer.disconnect();
  }
};

export default createNewOrder;

Inventory Service

This service module contains an Immediately Invoked Function Expression (IIFE) that continuously listens to the new-order-topic for new events. Upon receiving new order events, it processes and validates the products' inventory, updating the inventory status accordingly. Additionally, it sends events for notifications and logs actions related to inventory management.

import kafkaClient, {
  TOPIC_NEW_ORDER,
  TOPIC_INVENTORY_EMPTY,
  TOPIC_NEW_LOGGING_ACTION,
} from "../lib/kafka-client.js";
import productInventory from "../db/inventory.js";

// Initialize Kafka consumer and producer
const consumer = kafkaClient.consumer({ groupId: "inventory-service-group" });
const producer = kafkaClient.producer();

const sendEventToNotifyProductInventoryIsEmpty = async (productId) => {
  // Create a new event for 'inventory-empty-topic'
  await producer.send({
    topic: TOPIC_INVENTORY_EMPTY,
    messages: [{ value: JSON.stringify({ productId }) }],
  });
  // Create a new event for 'new-logging-action-topic'
  await producer.send({
    topic: TOPIC_NEW_LOGGING_ACTION,
    messages: [
      {
        value: JSON.stringify({
          serviceName: "inventory-service",
          action: "inventory-empty",
        }),
      },
    ],
  });
};

const processAndValidateProductsInventory = async (products) => {
  for (const product of products) {
    const prodQuantity = productInventory.product(product.productId);
    const updatedProdQuantity = prodQuantity - product.quantity;

    // Update product inventory
    productInventory.setProduct(product.productId, updatedProdQuantity);

    await producer.send({
      // create a new event for new-logging-action-topic
      topic: TOPIC_NEW_LOGGING_ACTION,
      messages: [
        {
          value: JSON.stringify({
            serviceName: "inventory-service",
            action: "inventory-updated",
          }),
        },
      ],
    });

    // Check if inventory is below threshold and take appropriate action
    if (updatedProdQuantity <= 100) {
      // Notify product inventory is low and log action
      await sendEventToNotifyProductInventoryIsEmpty(product.productId);

      // Refill product inventory
      productInventory.setProduct(product.productId, 109);
    }
  }
};

// IIFE: Listen for new events on 'new-order-topic'
(async () => {
  // Connect to Kafka broker
  await producer.connect();
  await consumer.connect();

  // Subscribe to 'new-order-topic'
  await consumer.subscribe({
    topics: [TOPIC_NEW_ORDER],
    // get all events from the beginning
    fromBeginning: true, // Start consuming from the beginning
  });

  // Run consumer
  await consumer.run({
    eachMessage: async ({ message }) => {
      const { products } = JSON.parse(message.value);

      await processAndValidateProductsInventory(products);
    },
  });
})();

Logging Service

This service module listens for events on the new-logging-action-topic and logs the received actions. It continuously runs, ensuring all relevant actions are logged as they occur.

import kafkaClient, { TOPIC_NEW_LOGGING_ACTION } from "../lib/kafka-client.js";

// Initialize Kafka consumer
const consumer = kafkaClient.consumer({ groupId: "logging-service-group" });

// IIFE: Listen for new events on 'new-logging-action-topic'
(async () => {
  // Connect to Kafka broker
  await consumer.connect();

  // Subscribe to 'new-logging-action-topic'
  await consumer.subscribe({
    topics: [TOPIC_NEW_LOGGING_ACTION],
    fromBeginning: true, // Start consuming from the beginning
  });

  // Run consumer
  await consumer.run({
    eachMessage: async ({ message }) => {
      const { serviceName, action } = JSON.parse(message.value);

      console.log(
        `\n--> Logging Service <--\nserviceName: ${serviceName}\naction: ${action}`
      );
    },
  });
})();

Notification Service

This service module listens for events on the new-order-topic and inventory-empty-topic and performs specific actions based on the received events. It dynamically calls the appropriate function based on the incoming event topic and logs the corresponding notification.

import kafkaClient, {
  TOPIC_NEW_ORDER,
  TOPIC_INVENTORY_EMPTY,
} from "../lib/kafka-client.js";

// Initialize Kafka consumer
const consumer = kafkaClient.consumer({
  groupId: "notification-service-group",
});

// an object of functions by topics to be called whenever an event arrives from that specific topic
const notificationHandlers = {
  [TOPIC_INVENTORY_EMPTY]: ({ productId }) =>
    console.log(
      `\n### Notification Service ###\nProduct ${productId} needs to be refilled.`
    ),
  [TOPIC_NEW_ORDER]: ({ orderId, customerId }) =>
    console.log(
      `\n### Notification Service ###\nCongrats ${customerId}, your order ${orderId} has been approved!`
    ),
};

// IIFE: Listen for new events on 'new-order-topic' and 'inventory-empty-topic'
(async () => {
  // Connect to Kafka broker
  await consumer.connect();

  // Subscribe to 'new-order-topic' and 'inventory-empty-topic'
  await consumer.subscribe({
    topics: [TOPIC_NEW_ORDER, TOPIC_INVENTORY_EMPTY],
    fromBeginning: true,
  });

  // Run consumer
  await consumer.run({
    eachMessage: async ({ topic, message }) => {
      // Call the appropriate notification handler based on the topic
      notificationHandlers[topic](JSON.parse(message.value));
    },
  });
})();

DB Module

This module contains a single class responsible for storing product quantities. It initializes with default product quantities and provides methods to retrieve and update product quantities.

class ProductInventory {
  #products;

  // Constructor initializes the products Map with default values
  constructor() {
    this.#products = new Map();
    this.#products.set("PRODUCT-123", 109);
    this.#products.set("PRODUCT-321", 109);
  }

  product(id) {
    return this.#products.get(id);
  }

  setProduct(id, value) {
    this.#products.set(id, value);
  }
}

// Export a singleton instance of ProductInventory
export default new ProductInventory();

Running the application

Start Kafka Broker: Open a second terminal and start the Kafka broker using the following command:

docker-compose up

Access Redpanda UI: Once the containers are up and running, you can access the Redpanda UI, an admin panel for the Kafka broker, by navigating to the following URL in your browser: http://127.0.0.1:8080

Start the Application: Return to the primary terminal and start the application using the start script:

npm start

View Logs: In the terminal, you will observe logs related to the Logging Service, as shown below:

Monitor Topics: In the Redpanda UI, navigate to the Topics menu to view the topics created and populated with events (messages):

Note: You can customize the orderOne object in the index.js file to observe different behaviors. Additionally, you can adjust the quantity in the db.js file.

That's it! You have successfully set up and run the application.

Happy coding! 🎉🎉🎉

Set up Hot Reload for Typescript ESM projects

Wagner Manganelli (aka manga) — Fri, 25 Aug 2023 19:38:24 +0000

TL;DR

Are you moving your projects to use ES Modules but you are facing some unexpected issues? You are not alone in this fight. In this tutorial, you will learn how to set up nodemon along with ts-node.

Wait, why are we not using ts-node-dev? Currently, there are some issues related to TS ESM projects that are making its use harder. Check this issue for more details.

Hot-Reload?

This technique is handy when you are developing locally. This will automatically watch changes over your source code and updates for you. So, you just have to make your code work and the hot-reloading will do the magic from behind the scenes. This will improve your code development process bringing more efficiency and velocity, as you will not have to restart every time your application.

Let's set it up 🔧

Prerequisites

Node.js >= v18.12.0

Initial steps

mkdir app-esm-hot-reload
cd app-esm-hot-reload
npm init -y

Install dev dependencies

npm install --save-dev typescript@5.1.6 ts-node@10.9.1 nodemon@3.0.1

Initialize TypeScript

  npx tsc --init

Configuration steps

package.json

{
  ...
  "type": "module",
  "scripts": {
    "build": "tsc",
    "dev": "nodemon src/index.ts",
    "start": "node dist/index.js"
  },
  ...
}

tsconfig.json

{
  "compilerOptions": {
    "target": "ESNext",
    "module": "NodeNext",
    "moduleResolution": "NodeNext",
    "outDir": "dist",
    "esModuleInterop": true,
    "strict": true,
    "skipLibCheck": true
  },
  "ts-node": {
    "esm": true
  },
  "exclude": ["node_modules", "./*.ts", "__test__"]
}

Create index.ts

mkdir src
touch src/index.ts

Execute script

npm run dev

Make changes to the index.ts or any other file

Your app should be reloaded automatically

That's it! Now you can start coding without bothering with restarting your application at every change. 🍺

P.S. you can refer to this template.

Useful links

Tutorials that might interest you as well

Set up a Node.js project + TypeScript + Jest using ES Modules

I hope this tutorial improves your development process.

Thank you for reading! Happy coding! 🥳

Set up a Node.js project + TypeScript + Jest using ES Modules

Wagner Manganelli (aka manga) — Thu, 24 Aug 2023 20:04:45 +0000

TL;DR

In this tutorial, you will learn how to configure your Node.js app with TypeScript + Jest using ECMAScript modules (ESM).

What are ECMAScript modules?

ECMAScript modules are the official standard format to package JavaScript code for reuse. Modules are defined using a variety of import and export statements. - nodejs.org

As the description says, it is the official standard format. To know more about the differences and benefits there are plenty of articles talking about it. The intention here is to show you how to set up as smoothly as possible your app.

Let's set it up 🔧

Prerequisites

Node.js >= v18.12.0

Initial steps

mkdir app-esm
cd app-esm
npm init -y

Install dev dependencies

npm install --save-dev typescript@5.1.6 ts-node@10.9.1 ts-jest@29.1.1 jest@29.6.2 @types/jest@29.5.3

Initialize TypeScript

npx tsc --init

Set up project folder structure

app-esm/
├── __test__/
│   └── index.test.ts
├── src/
│   └── index.ts
├── jest.config.ts
├── package.json
└── tsconfig.json

P.S. create manually jest.config.ts

Configuration steps

package.json

{
  ...
  "type": "module",
  "scripts": {
    "build": "tsc",
    "test": "node --experimental-vm-modules node_modules/.bin/jest",
    "start": "node dist/index.js"
  },
  ...
}

tsconfig.json

{
  "compilerOptions": {
    "target": "ESNext",
    "module": "NodeNext",
    "moduleResolution": "NodeNext",
    "outDir": "dist",
    "esModuleInterop": true,
    "strict": true,
    "skipLibCheck": true
  },
  "exclude": ["node_modules", "./*.ts", "__test__"]
}

jest.config.ts

import type { JestConfigWithTsJest } from "ts-jest";

const config: JestConfigWithTsJest = {
  verbose: true,
  transform: {
    "^.+\\.ts?$": [
      "ts-jest",
      {
        useESM: true,
      },
    ],
  },
  extensionsToTreatAsEsm: [".ts"],
  moduleNameMapper: {
    "^(\\.{1,2}/.*)\\.js$": "$1",
  },
};

export default config;

That's it! Now you can start coding without bothering with the setup part. 🍺

P.S. you can refer to this template. There you will find some jest tests with mock examples.

Useful links

Tutorials that might interest you as well

Set up Hot Reload for Typescript ESM projects

I hope this tutorial helps you focus on what really matters.

Thank you for reading! Happy coding! 🥳