DEV Community: Michael Tyiska

Setting Up Machine Learning Pipelines with GitOps Principles

Michael Tyiska — Sat, 07 Dec 2024 23:21:06 +0000

In the ever-evolving world of DevOps and machine learning, building scalable and automated workflows has become a cornerstone for success. This guide walks you through setting up an end-to-end MLOps pipeline using GitOps principles, incorporating tools like Argo Workflows, Argo Events, MinIO, FastAPI, MLflow, Kubernetes, and Evidently AI. By following this setup, you’ll have a robust system to detect data drift, retrain models, and deploy them seamlessly.

Architecture Overview

This architecture is designed to handle the entire lifecycle of a machine learning model, ensuring it stays accurate and reliable over time.

Pre-Deployment

We start with Data Preparation, where raw datasets are cleaned and split into training and testing sets. Next, in Model Development, we train and evaluate multiple models using MLflow to identify the best-performing one. Finally, the best model, along with the processed datasets and reference files, is stored in MinIO for version control and easy retrieval.

CI/CD Integration

The pipeline incorporates continuous integration and deployment principles to ensure that changes to the system, whether in the codebase or data, are quickly and safely integrated. This includes linting, testing, and automated deployment steps.

Deployment

The trained model is deployed via a lightweight FastAPI application, which serves predictions and continuously monitors incoming data for drift. This ensures that the model remains accessible and up-to-date.

Post-Deployment

Using Evidently AI, the system monitors the production data for drift. If significant drift is detected, a Kubernetes CronJob triggers the retraining process. The retraining uses the updated data, and the new model is saved back to MinIO. The FastAPI app then dynamically reloads the updated model, enabling seamless updates without manual intervention.

Feedback Loop

This workflow creates a fully automated feedback loop: it detects drift, retrains models, and redeploys them, ensuring that the system remains reliable and accurate over time.

Here’s a high-level diagram of the architecture:

Getting Started

Environment Setup

This section focuses on preparing your Python environment for the MLOps pipeline. By using tools like conda and pip, we ensure that all dependencies for retraining models, deploying applications, and development are properly installed. Setting up a clean environment is critical to avoid conflicts between packages and to maintain reproducibility across systems. The MLflow server, a key component, is used to log and track your machine learning experiments in a centralized way.

Start by setting up your Python environment and installing dependencies:

conda create --name mlops_env python=3.11 -y
conda activate mlops_env
pip install -r requirements/requirements.retrain.txt --force-reinstall
pip install -r requirements/requirements.app.txt
pip install -r requirements/requirements.dev.txt

Launch the MLflow server to track experiments:

mlflow server --host 127.0.0.1 --port 8080

Makefile Commands

Makefiles streamline repetitive tasks by defining commands that can be run consistently with a single line. This section highlights common operations like linting, testing, data preparation, and model training. Each Makefile command corresponds to a specific part of the workflow, making it easy to execute complex tasks without manually typing long commands.

Here are the key commands:

Linting and Testing

   make lint
   make test

Data Preparation

   make run-data-preparation

Model Training

   make run-model-training

Model Evaluation

   make run-evaluate

Model Retraining

   make run-retrain

Run FastAPI Locally

   make run-fastapi

Access the API documentation:

http://127.0.0.1:8000/docs

Setting Up Kubernetes

Kubernetes is a container orchestration platform that allows you to deploy, manage, and scale services. This section guides you through creating namespaces, which provide logical isolation for different services like MLflow, MinIO, and FastAPI. Using namespaces ensures that each service runs independently, simplifying resource management and troubleshooting.

Namespace Creation

Create separate namespaces for each service:

kubectl create namespace mlserver
kubectl create namespace minio
kubectl create namespace mlflow
kubectl create namespace fastapi
kubectl create namespace argo
kubectl create namespace argo-events

MinIO Setup

MinIO is an object storage system compatible with AWS S3. It is used in this setup to store large datasets, models, and other artifacts. Deploying MinIO involves setting up services and ingress configurations to enable access. Exposing MinIO services allows interaction with the storage directly, making it a critical part of the data pipeline.

Deploy MinIO for object storage:

kubectl apply -f minio_depl.yml
kubectl apply -f minio-ingress.yaml

Expose MinIO services:

kubectl port-forward svc/minio-service 9000:9000 -n minio

Deploy MLServer

MLServer serves machine learning models and provides an interface for inference requests. This section outlines how to build a Docker image for MLServer, push it to a container registry, and deploy it on Kubernetes. By port-forwarding the service, you can access MLServer locally, making it easier to test and debug deployments.

Build and push the MLServer image:

docker build --platform linux/amd64 -t measureapp/mlserver:0.0.2 .
docker push measureapp/mlserver:0.0.2

Apply the deployment:

kubectl apply -f mlserver.yaml
kubectl get pods -n mlserver

Access the service:

kubectl port-forward svc/mlserver-service 5000:5000 -n mlserver

Steps to Set Up MLServer

This section expands on deploying MLServer by detailing additional steps like creating a Docker registry secret for pulling container images securely. It also covers verifying the deployed MLServer instance and accessing the MLflow UI for monitoring workflows within the pod. This ensures the MLServer is correctly configured and running.

Build and Push MLServer Docker Image:

   docker build --platform linux/amd64 -t measureapp/mlserver:0.0.2 .
   docker push measureapp/mlserver:0.0.2

Create Docker Registry Secret:

   kubectl create secret -n mlserver docker-registry docker-registry-creds \
       --docker-server=https://index.docker.io/v1/ \
       --docker-username=measureapp --docker-password=sensitive_password

Deploy MLServer:

   kubectl apply -f mlserver.yaml

Verify MLServer Pods:

   kubectl get pods -n mlserver

Access MLServer via Port Forwarding:

Forward the service port:

 kubectl port-forward svc/mlserver-service 5000:5000 -n mlserver

Open the MLflow UI in your browser:
```
 http://localhost:5000
```

Alternative Port Forwarding:

   kubectl port-forward svc/mlserver-service 8080:8080 -n mlserver

Verify MLflow Process Inside MLServer Pod:

Execute into the pod:

 kubectl exec -it mlserver-<pod-name> -n mlserver -- bash

Check running processes:

 ps aux | grep mlflow
 ps aux | grep mlserver

Steps to Set Up Model Job

The model job refers to the training and evaluation pipeline for machine learning models. This section discusses building a containerized job, securely managing container images via Kubernetes secrets, and deploying the job on the cluster. It ensures that model training runs reliably and integrates well with other components.

Create Docker Registry Secret for MLflow Namespace:

   kubectl create secret -n mlflow docker-registry docker-registry-creds \
       --docker-server=https://index.docker.io/v1/ \
       --docker-username=measureapp --docker-password=passwordbuild

Build and Push Model Job Docker Image:

   docker build --platform linux/amd64 -t measureapp/ml_job:0.0.26 -f docker/model/Dockerfile .
   docker push measureapp/ml_job:0.0.26

Deploy Model Job:

   kubectl apply -f model-train-job.yaml

Verify Model Job Pods:

   kubectl -n mlflow get pods

Steps to Set Up FastAPI

FastAPI is used to serve the API endpoints for interacting with the machine learning pipeline. This section explains setting up Docker registry credentials, building and deploying FastAPI as a container, and accessing the service locally. FastAPI provides a user-friendly interface for interacting with models and simulating various scenarios.

Create Docker Registry Secret for FastAPI Namespace:

   kubectl create secret -n fastapi docker-registry docker-registry-creds \
       --docker-server=https://index.docker.io/v1/ \
       --docker-username=measureapp --docker-password=sensitive_password

Build and Push FastAPI Docker Image:

   docker build --platform linux/amd64 -t measureapp/demo_ai_api:0.0.40 -f docker/fastapi/Dockerfile .
   docker push measureapp/demo_ai_api:0.0.40

Deploy FastAPI:

   kubectl apply -f fastapi-depl.yaml

Verify FastAPI Pods:

   kubectl -n fastapi get pods

Access FastAPI Service via Port Forwarding:

   kubectl -n fastapi port-forward svc/fastapi-service 8000:80

Open in browser:

   http://localhost:8000

Steps to Set Up Argo Workflows and Events

Argo Workflows orchestrate complex workflows in Kubernetes. This section describes setting up Argo for managing pipeline automation and integrating Argo Events for triggering workflows based on specific events like data drift. The integration ensures your pipeline is dynamic and responsive to changes.

Install Argo Workflows

Set Argo Workflows Version:

   ARGO_WORKFLOWS_VERSION="v3.6.0"

Install Argo Workflows:

   kubectl apply -n argo -f "https://github.com/argoproj/argo-workflows/releases/download/${ARGO_WORKFLOWS_VERSION}/quick-start-minimal.yaml"

Verify Installation:

   kubectl get all -n argo

Wait for Pods to Be Ready.

Access Argo Server:

Port forward the Argo server:

 kubectl -n argo port-forward service/argo-server 2746:2746

Open the UI in your browser:
```
 https://localhost:2746
```

Create Secrets

GitHub Credentials for Argo:

   kubectl create secret generic git-credentials \
       --from-literal=username=mtyiska \
       --from-literal=token=sensitive_token -n argo

GitHub Token for Argo Events:

   kubectl create secret generic github-token-secret \
       --from-literal=token=sensitive_token \
       --namespace=argo-events

Docker Registry Credentials for Argo:

   kubectl create secret -n argo docker-registry docker-registry-creds \
       --docker-server=https://index.docker.io/v1/ \
       --docker-username=measureapp --docker-password=sensitive_password

GitHub Credentials for Argo Events:

   kubectl create secret generic git-credentials \
       --from-literal=username=mtyiska \
       --from-literal=token=sensitive_token -n argo-events

Docker Registry Credentials for Argo Events:

   kubectl create secret -n argo-events docker-registry docker-registry-creds \
       --docker-server=https://index.docker.io/v1/ \
       --docker-username=measureapp --docker-password=sensitive_password

Install Argo Events

Drift detection is essential for monitoring model performance over time. This section explains how to set up workflows and events to detect and handle drift automatically. By simulating drift through FastAPI or triggering events manually, you can validate the robustness of the drift handling pipeline.

Install Argo Events:

   kubectl apply -f https://raw.githubusercontent.com/argoproj/argo-events/stable/manifests/install.yaml

Install Validating Webhook:

   kubectl apply -f https://raw.githubusercontent.com/argoproj/argo-events/stable/manifests/install-validating-webhook.yaml

Deploy Native EventBus:

   kubectl apply -n argo-events -f https://raw.githubusercontent.com/argoproj/argo-events/stable/examples/eventbus/native.yaml

Create RBAC Policies:

For sensors:

 kubectl apply -n argo-events -f https://raw.githubusercontent.com/argoproj/argo-events/master/examples/rbac/sensor-rbac.yaml

For workflows:

 kubectl apply -n argo-events -f https://raw.githubusercontent.com/argoproj/argo-events/master/examples/rbac/workflow-rbac.yaml

Verify Installation:

   kubectl get all -n argo-events

Wait for Pods to Be Ready.
Verify Default Service Account:

   kubectl -n argo-events describe serviceaccount default

Apply Patch:
- Navigate to the serviceaccount directory:
```
 cd serviceaccount
```

Apply the patch:
```
 kubectl apply -f .
```

Verify Service Account:

   kubectl -n argo-events describe serviceaccount default

Drift Workflow and Events

This section focuses on creating and deploying a drift detection job. Drift detection jobs analyze incoming data for changes that might impact model performance, ensuring that models remain accurate and reliable. It highlights the importance of monitoring changes in data distribution.

Navigate to Drift Workflow Directory:

   cd argo

Apply Drift Workflow and Events:

   kubectl apply -f .

Build and Deploy Drift Detection Job

Build Drift Detection Docker Image:

   docker build --platform linux/amd64 -t measureapp/drift_detection:0.0.4 -f docker/drift-detection/Dockerfile .

Push Drift Detection Docker Image:

   docker push measureapp/drift_detection:0.0.4

Deploy Drift Detection Job:

   kubectl apply -f drift-job.yaml

Verify Pods:

   kubectl -n mlflow get pods

Access Argo Server

Accessing the Argo server enables you to view and manage workflow executions. This section explains how to connect to the Argo UI, where you can visualize workflows, monitor their progress, and troubleshoot issues. The UI provides insights into the automation and health of your pipeline.

Port Forward Argo Server:

   kubectl -n argo port-forward service/argo-server 2746:2746

Access UI:

   https://localhost:2746

Manually Submit the Workflow

Manual workflow submission helps test the pipeline and ensure all configurations are correct. This section explains how to trigger workflows directly from the Argo CLI, which is helpful during development and debugging. Monitoring execution logs provides visibility into each step, ensuring workflows run as expected.

Submit Workflow Using Argo CLI:

   argo submit --from workflowtemplate/drift-retrain-template \
       -p data-dir="s3://minio/data/app/data/processed" \
       -p model-dir="s3://minio/models/best_model" \
       -p reference-data-path="s3://minio/data/app/data/processed/X_train.csv" \
       -p current-data-path="s3://minio/data/app/data/processed/X_test.csv" \
       -n argo-events

Monitor Workflow Execution:

   kubectl logs -n argo-events -f $(kubectl get pods -n mlflow --selector=workflows.argoproj.io/workflow=<WORKFLOW-NAME> -o jsonpath='{.items[0].metadata.name}')

Test and Simulate Drift

Testing drift scenarios is crucial for validating your pipeline's ability to handle changes in data. This section explains how to simulate drift using FastAPI endpoints and monitor logs to verify that drift events are detected and processed correctly. This step ensures your pipeline is prepared for real-world data challenges.

Trigger Drift Event:

   kubectl exec -it fastapi-deployment-555f9fd4cb-sbv9z -n fastapi -- curl -X POST \
       -H "Content-Type: application/json" \
       -d '{"drift_score": 0.5}' \
       http://drift-detection-eventsource-svc.argo-events.svc.cluster.local:12000/drift-detected

Check Event Logs:

   kubectl get pods -n argo-events
   kubectl -n argo-events logs drift-detection-eventsource-lgtsv-6f949fdcf7-znvh2
   kubectl -n argo-events logs drift-detection-sensor-sensor-kwdvx-bbbc5db57-qd659

Simulate Drift via FastAPI:

   curl -X POST "http://localhost:8000/simulate-drift" \
       -H "Content-Type: application/json" \
       -d '{"drift_type": "numerical_shift"}'

Final Thoughts

This guide demonstrates how to build an automated pipeline for model drift detection and retraining using GitOps principles. By combining Kubernetes-native tools like Argo Workflows, Events, and MinIO, you can ensure your machine learning workflows are scalable, reliable, and efficient.

Check out the complete repository here:

GitHub Repository Link

If you enjoyed this article or have questions, feel free to connect with me on LinkedIn or drop a comment below. Let’s build smarter, together! 🚀

Automating CI/CD Pipelines for Kubernetes with Argo Rollouts, Argo CD, Argo Workflow & Events

Michael Tyiska — Sat, 30 Nov 2024 22:37:09 +0000

As DevOps continues to evolve, GitOps has emerged as a key practice for managing Kubernetes deployments. By treating your Git repository as the single source of truth, you can ensure reliable, repeatable, and automated deployments. This blog will walk you through a complete infrastructure setup and deployment pipeline for a Node.js API using Argo Rollouts, Argo CD, and Argo Events.

Find the complete code and configuration for this setup on GitHub:
GitHub Repository Link Infra

GitHub Repository Link Node API Example

🌟 Let’s Connect!
I love sharing insights about DevOps, Kubernetes, and GitOps tools like ArgoCD. If you found this article helpful or have questions, let’s continue the conversation on LinkedIn!
👉 Connect with me on LinkedIn

Why Argo Rollouts?

Traditional Kubernetes deployments are robust, but they lack built-in support for progressive delivery strategies. Argo Rollouts fills this gap by providing:

Blue-Green Deployments: Seamlessly shift traffic between environments.
Canary Rollouts: Incrementally release features to minimize risk.
Real-Time Monitoring: Observe application health during rollouts.

By incorporating Argo Rollouts into our workflow, we can enhance the reliability of Kubernetes-based applications while maintaining GitOps principles.

Part 1: Infrastructure & Rollout Strategy

Installing Kind

Kind is essential for creating local Kubernetes clusters for testing. Install it using:

brew install kind

Installing Argo Rollouts CLI

The Argo Rollouts plugin manages blue-green and canary deployments. Install it based on your OS:

macOS:

curl -LO https://github.com/argoproj/argo-rollouts/releases/latest/download/kubectl-argo-rollouts-darwin-arm64
chmod +x ./kubectl-argo-rollouts-darwin-arm64
sudo mv ./kubectl-argo-rollouts-darwin-arm64 /usr/local/bin/kubectl-argo-rollouts

Linux:

curl -LO https://github.com/argoproj/argo-rollouts/releases/latest/download/kubectl-argo-rollouts-linux-amd64
chmod +x ./kubectl-argo-rollouts-linux-amd64
sudo mv ./kubectl-argo-rollouts-linux-amd64 /usr/local/bin/kubectl-argo-rollouts

Creating Multi-Node Clusters

Set the Kubernetes configuration file:
```
export KUBECONFIG=$PWD/kubeconfig.yaml
```

Create a cluster using Kind:

kind create cluster --name argo-cluster --config kind-config.yaml

Verify the cluster nodes:
```
kubectl get nodes
```

Setting Up NGINX Ingress Controller

Deploy an ingress controller for managing HTTP traffic:

helm upgrade --install ingress-nginx ingress-nginx \
  --repo https://kubernetes.github.io/ingress-nginx \
  --namespace ingress-nginx --create-namespace \
  --set controller.hostPort.enabled=true \
  --set controller.service.type=NodePort \
  --set controller.hostPort.ports.http=80 \
  --set-string controller.nodeSelector."kubernetes\.io/os"=linux \
  --set-string controller.nodeSelector.ingress-ready="true"

Label the worker node:

kubectl label node argo-cluster-worker ingress-ready=true

Verify the ingress controller:

kubectl get pods -n ingress-nginx

Setting Up Argo Rollouts

Create a namespace for Argo Rollouts:
```
kubectl create namespace argo-rollouts
```

Download and apply the Argo Rollouts controller:

curl -L -o argo-controller.yaml https://github.com/argoproj/argo-rollouts/releases/latest/download/install.yaml
kubectl apply -n argo-rollouts -f argo-controller.yaml

Verify the installation:

kubectl api-resources | grep -i argo
kubectl argo rollouts version

Docker Secrets for Kubernetes

Generate Docker Hub access tokens for private image pulls:

Create Docker secrets for the staging namespace:

kubectl create secret -n staging docker-registry docker-registry-creds \
   --docker-server=https://index.docker.io/v1/ \
   --docker-username=<username> --docker-password=<password>

Repeat for the production namespace.

```bash
kubectl create secret -n prod docker-registry docker-registry-creds \
   --docker-server=https://index.docker.io/v1/ \
   --docker-username=<username> --docker-password=<password>
```

Deploying Staging with Blue-Green Strategy

Change directory to the application’s kustomize folder:
```
cd apps/node-demo/
```
Apply the deployment:
```
kubectl apply -k staging
```

Verify the deployment:

kubectl -n staging get all
kubectl -n staging get ingress

Update /etc/hosts:

echo "127.0.0.1 staging.node-api.example.com" | sudo tee -a /etc/hosts

Test the API:
- Swagger UI: http://staging.node-api.example.com/swagger/
- Preview: http://staging.node-api.example.com/swagger/preview

Watch rollouts:

kubectl argo rollouts dashboard -p 3100

open a new terminal (terminal 2) and run this command:

kind export kubeconfig --name argo-cluster

Deploying Production with Canary Strategy

Change directory:
```
cd apps/node-demo/
```
Deploy the application to production:
```
kubectl apply -k prod
```

Verify production deployment:

kubectl -n prod get all
kubectl -n prod get ingress

Update /etc/hosts:

echo "127.0.0.1 prod.node-api.example.com" | sudo tee -a /etc/hosts

Access the production API:

- `http://prod.node-api.example.com/swagger`
- `http://prod.node-api.example.com/swagger/preview`

Test the Rollout with a Patch:

TIMESTAMP=$(date +'%Y%m%d%H%M%S')
kubectl -n staging patch rollout node-api --type='json' -p="[{
    \"op\": \"add\",
    \"path\": \"/spec/template/metadata/annotations\",
    \"value\": {\"force-update\": \"$TIMESTAMP\"}
}]"

Setting Up Continuous Deployment with Argo CD

In this part we’ll focus on integrating Argo CD, a powerful continuous deployment tool that allows you to automate and sync your Kubernetes applications directly from Git repositories. This step ensures your deployment pipelines are seamless, automated, and fully aligned with GitOps principles.

Why Argo CD?

Argo CD enhances your deployment process by providing:

GitOps Automation: Automatically sync your Kubernetes clusters with your Git repository.
Declarative Configurations: Manage everything via manifests stored in Git.
Rollbacks: Easily roll back to previous application states.
Visibility: Monitor deployment health through a user-friendly UI and CLI.

This makes Argo CD an ideal choice for handling Kubernetes workloads at scale.

Step 1: Install Argo CD

to avoid any confusion, we're still in terminal 2, so cd back to the root of the folder

To start, install Argo CD on your Kubernetes cluster.

Create a namespace for Argo CD:
```
kubectl create namespace argocd
```

Apply the Argo CD manifest:

kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

Verify the Argo CD components:
```
kubectl get all -n argocd
```

Wait until all the components are in the Running state.

Step 2: Configure Password & Server Access

Next, secure the Argo CD server with a custom admin password.

Generate a secure password and hash:

PASSWORD=$(openssl rand -base64 16)
BCRYPT_HASH=$(htpasswd -bnBC 10 "" "$PASSWORD" | tr -d ':\n')

Set the password in the Argo CD secret:

kubectl -n argocd patch secret argocd-secret \
  -p '{"stringData": {
    "admin.password": "'"$BCRYPT_HASH"'",
    "admin.passwordMtime": "'$(date +%FT%T%Z)'"
  }}'

Restart the Argo CD server to apply changes:

kubectl -n argocd rollout restart deployment argocd-server

Expose the server using NodePort:

kubectl patch svc argocd-server -n argocd --patch '{"spec": {"type": "NodePort", "ports": [{"nodePort": 32100, "port": 443, "protocol": "TCP", "targetPort": 8080}]}}'

Access the Argo CD UI:

- Forward the port:

    kubectl port-forward svc/argocd-server 8080:80 -n argocd

- Open in your browser: `http://localhost:8080`

in terminal 3 run this command

kind export kubeconfig --name argo-cluster

Step 3: Install and Configure the Argo CD CLI

To interact with Argo CD via the CLI:

Download the CLI:

sudo curl -sSL -o /usr/local/bin/argocd https://github.com/argoproj/argo-cd/releases/latest/download/argocd-darwin-amd64

Make it executable:
```
sudo chmod +x /usr/local/bin/argocd
```
Login to Argo CD:
```
argocd login localhost:8080
```
Verify the CLI version:
```
argocd version
```

Step 4: Integrate Git with Argo CD

To sync repositories with Argo CD, configure SSH keys for secure access.

Step 4.1: Create a New SSH Key

Generate an SSH key pair:

ssh-keygen -t ed25519 -C "your_email@example.com" -f ~/.ssh/new_github_key

- Replace `your_email@example.com` with your GitHub email.

Add the public key to GitHub:

- Copy the public key:

    ```bash
    cat ~/.ssh/new_github_key.pub
    ```

- Go to **GitHub > Settings > SSH and GPG keys > New SSH key**, and paste the key.

Step 4.2: Configure the SSH Key for GitHub

Edit your SSH config:

nano ~/.ssh/config

Add:

Host github-new
    HostName github.com
    User git
    IdentityFile ~/.ssh/new_github_key

Test the SSH connection:

ssh -T github-new

You should see:

Hi <username>! You've successfully authenticated, but GitHub does not provide shell access.

Step 4.3: Add the Repository to Argo CD

Add the repository:

argocd repo add git@github.com:your-username/your-repo.git \
  --ssh-private-key-path ~/.ssh/new_github_key \
  --name argo-boot

Verify the repository:
```
argocd repo list
```

Step 5: Deploy Applications Using Argo CD

Ensure the Github repos are created and create a release branch as it is needed.

Navigate to your Argo CD project:
```
cd infra/argo-cd/projects/
```
Apply the project YAML:
```
kubectl apply -f node-demo.yaml
```

Deploy staging and production applications:

cd ../applications
kubectl apply -f node-demo-staging.yaml
kubectl apply -f node-demo-prod.yaml

Verify deployments in the Argo CD UI:

- Open: `http://localhost:8080/applications`

Setting Up Continuous Integration with Argo Workflows

In this part we’ll focus on building a Continuous Integration (CI) pipeline using Argo Workflows and Argo Events. These tools form the backbone of an automated system that responds to changes in your Git repositories, builds your application, and prepares it for deployment.

Why Argo Workflows and Argo Events?

Argo Workflows: A Kubernetes-native workflow engine designed for large-scale automation. Perfect for tasks like building Docker images or running tests.
Argo Events: A Kubernetes event-based dependency manager that triggers workflows based on events like Git pushes or webhooks.

Together, these tools automate CI pipelines with real-time event-driven capabilities.

cd back to the root folder

Step 1: Install Argo Workflows

1. Create a namespace for Argo Workflows:

kubectl create namespace argo

2. Apply the Argo Workflows manifest:

ARGO_WORKFLOWS_VERSION="v3.6.0"
kubectl apply -n argo -f "https://github.com/argoproj/argo-workflows/releases/download/${ARGO_WORKFLOWS_VERSION}/quick-start-minimal.yaml"

3. Verify the installation:

kubectl get all -n argo

Wait until all components are in the Running state.

4. Port-forward to access the Argo Workflows UI:

kubectl -n argo port-forward service/argo-server 2746:2746

Access the Argo Workflows UI in your browser:

https://localhost:2746

now in the final terminal, terminal 4, run this command

kind export kubeconfig --name argo-cluster

Step 2: Install and Configure the Argo CLI

Set your system architecture:

ARGO_OS="darwin"  # macOS users
ARGO_ARCH="amd64" # Default architecture

# Adjust for M1 (ARM-based architecture):
if [[ $(uname -m) == "arm64" ]]; then
  ARGO_ARCH="arm64"
fi

Download the Argo CLI:

curl -sLO "https://github.com/argoproj/argo-workflows/releases/download/v3.6.0/argo-$ARGO_OS-$ARGO_ARCH.gz"

Unzip the binary:
```
gunzip "argo-$ARGO_OS-$ARGO_ARCH.gz"
```
Make the binary executable:
```
chmod +x "argo-$ARGO_OS-$ARGO_ARCH"
```

Move the binary to your system path:

sudo mv "./argo-$ARGO_OS-$ARGO_ARCH" /usr/local/bin/argo

Verify the installation:
```
argo version
```

Step 3: Configure GitHub Integration

Generate a Personal Access Token (PAT)

Go to GitHub Personal Access Tokens.
Click Generate new token and select:
- repo for private repository access.
- Optionally, read:org for organizational repositories.
Copy the token once it’s generated.

Create Kubernetes Secrets for GitHub Integration

if you run into any issues, make sure all 4 secrets are created

Create a namespace for Argo Events:
```
kubectl create namespace argo-events
```

Add your GitHub credentials as a secret:

kubectl create secret generic git-credentials \
  --from-literal=username=<your-username> \
  --from-literal=token=<your-github-token> \
  -n argo

For events, create a GitHub token secret:

kubectl create secret generic github-token-secret \
  --from-literal=token=<your-github-token> \
  --namespace=argo-events

Add Docker credentials:

kubectl create secret -n argo docker-registry docker-registry-creds \
  --docker-server=https://index.docker.io/v1/ \
  --docker-username=<docker-username> \
  --docker-password=<docker-password>

Add Github credentials for argo-events (different than above)

kubectl create secret generic git-credentials \
  --from-literal=username=mtyiska \
  --from-literal=token=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  -n argo-events

Step 4: Set Up Argo Events

Install Argo Events

Apply the Argo Events manifests:

kubectl apply -f https://raw.githubusercontent.com/argoproj/argo-events/stable/manifests/install.yaml 
kubectl apply -f https://raw.githubusercontent.com/argoproj/argo-events/stable/manifests/install-validating-webhook.yaml 
kubectl apply -n argo-events -f https://raw.githubusercontent.com/argoproj/argo-events/stable/examples/eventbus/native.yaml

Create RBAC Policies:

kubectl apply -n argo-events -f https://raw.githubusercontent.com/argoproj/argo-events/master/examples/rbac/sensor-rbac.yaml 
kubectl apply -n argo-events -f https://raw.githubusercontent.com/argoproj/argo-events/master/examples/rbac/workflow-rbac.yaml

Verify Argo Events component:
```
kubectl get all -n argo-events
```

Wait until all components are ready.

Step 5: Configure Event Sources and Sensors

Navigate to the event sources directory:
```
cd infra/argo-events/event-sources
```

Apply the webhook event source:

kubectl apply -f webhook-eventsource.yaml

Move to the pipeline templates:
```
cd ../../../pipelines/templates
```

Deploy the CI template:

kubectl apply -f node-api-ci-template.yaml

Navigate to the sensors directory:
```
cd ../../infra/argo-events/sensors
```
Apply the polling sensor:
```
kubectl apply -f polling-sensor.yaml
```

Step 6: Build and Deploy Custom Poller

Navigate to the custom poller directory:
```
cd ../custom-poller
```

Build and push the poller image:

docker build --platform linux/amd64 -t <your-docker-registry>/github-poller:0.0.1 .

Deploy the poller:
```
kubectl apply -f poller-cronjob.yaml
```

Test the workflow

To test the changes make a simple change in the demo nodejs app. You can change the Version in the index.ts as it doesn't modify the code but will still trigger the workflow

Wrapping Up

You’ve now set up Continuous Integration (CI) using Argo Workflows and Argo Events. You can view workflows in the Argo Workflows UI by visiting:

https://localhost:2746/workflows/argo-events?&limit=50