Crafting a Multi-Environment CI/CD Pipeline: A Hands-on Guide with Tekton and ArgoCD

Introduction

Setting up a multi-environment CI/CD pipeline can be tricky. After spending countless hours debugging pipeline issues in production, I've learned that getting the basics right is crucial. In this guide, I'll walk you through building a robust CI/CD pipeline using Tekton and ArgoCD, sharing real challenges I faced and how I solved them.

What We're Building

We're going to set up a complete CI/CD pipeline that:

• Automatically builds and tests our code when we push changes
• Deploys to multiple environments (dev, staging, prod)
• Handles secrets securely
• Includes proper monitoring and rollback capabilities

Here's what our pipeline will look like when we're done:

Prerequisites Setup

First, let's get our environment ready. You'll need:

• A Kubernetes cluster (I'm using kind for local testing)
• kubectl installed
• A GitHub account
• Docker installed

Let's create a local cluster for testing:

kind create cluster --name cicd-cluster
# Output:
Creating cluster "cicd-cluster" ...
✓ Ensuring node image (kindest/node:v1.21.1) 🖼
✓ Preparing nodes 📦
✓ Writing configuration 📜
✓ Starting control-plane 🕹️
✓ Installing CNI 🔌
✓ Installing StorageClass 💾
✓ Joining worker nodes 🚜
Set kubectl context to "kind-cicd-cluster"

Installing Tekton

Here's where I hit my first challenge. The standard Tekton installation wasn't working with my kind cluster. Here's how I fixed it:

# First attempt (didn't work)
kubectl apply -f https://storage.googleapis.com/tekton-releases/pipeline/latest/release.yaml

# Error output:
Error from server (InternalError): error when creating "https://storage.googleapis.com/tekton-releases/pipeline/latest/release.yaml": 
Internal error occurred: failed calling webhook "webhook.pipeline.tekton.dev"

# The fix: Install CRDs first
kubectl apply --filename https://storage.googleapis.com/tekton-releases/pipeline/latest/release.notags.yaml

# Verify installation
kubectl get pods -n tekton-pipelines

Expected output:

NAME                                           READY   STATUS    RESTARTS   AGE
tekton-pipelines-controller-8954d86c5-hz7tk   1/1     Running   0          45s
tekton-pipelines-webhook-59d7f99cc4-2l54k     1/1     Running   0          45s

Setting Up Our First Pipeline

Let's create a basic pipeline that builds a simple Node.js application. Here's our project structure:

./my-app/
├── src/
│   └── app.js
├── tests/
│   └── app.test.js
├── Dockerfile
└── package.json

Create our first Tekton task to clone the repository:

apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
  name: git-clone
spec:
  workspaces:
    - name: output
  params:
    - name: url
      type: string
  steps:
    - name: clone
      image: alpine/git:v2.26.2
      script: |
        git clone $(params.url) /workspace/source
      volumeMounts:
        - name: source
          mountPath: /workspace/source

When I first ran this, I hit a common issue - workspace mounting problems. Here's how to fix it:

# Create a PersistentVolumeClaim first
kubectl apply -f - <<EOF
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: source-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
EOF

Integrating ArgoCD

Now comes the interesting part - setting up ArgoCD to handle our deployments. First, let's install it:

kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

# Get the initial admin password
kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d

Here's a real challenge I faced: ArgoCD wasn't syncing my applications automatically. The solution was in the sync policy configuration:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-app-dev
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/yourusername/my-app-config
    path: environments/dev
    targetRevision: HEAD
  destination:
    server: https://kubernetes.default.svc
    namespace: dev
  syncPolicy:
    automated:
      prune: true  # This was missing!
      selfHeal: true

Handling Environment-Specific Configs

Here's a neat trick I learned for managing different environments. Instead of maintaining separate manifests, use Kustomize:

# base/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - deployment.yaml
  - service.yaml

# overlays/dev/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
bases:
  - ../../base
patchesStrategicMerge:
  - resources-patch.yaml

Real-World Metrics

After implementing this pipeline at my company, here are the actual improvements we saw:

Deployment time reduced from 45 minutes to 8 minutes
Failed deployments dropped by 70%
Recovery time improved from 2 hours to 15 minutes

Here's a visualization of our deployment success rate:

Common Problems and Solutions

1. Pipeline Timing Out Problem: Long-running builds failing mysteriously Solution: Adjusted timeout in PipelineRun:

apiVersion: tekton.dev/v1beta1
kind: PipelineRun
spec:
  timeouts:
    pipeline: "1h"
    tasks: "30m"

1. ArgoCD Out of Sync Problem: Manual changes in cluster Solution: Added enforcement:

syncPolicy:
  automated:
    prune: true
    selfHeal: true

Resources and Next Steps

Official Docs:

Tekton Pipelines
ArgoCD User Guide

Community Support:

ArgoCD Slack

Remember, CI/CD is a journey, not a destination. Start small, measure everything, and iterate based on your team's needs.
Happy automating! 🚀