DEV Community

Jamil Shaikh
Jamil Shaikh

Posted on

🚀 Building a GitOps Infrastructure Pipeline with Crossplane and Argo CD

#kubernetes #argocd #linux #crossplane

From manual kubectl commands to fully automated infrastructure management - here's how I built a production-ready GitOps pipeline

TL;DR

I built a complete GitOps infrastructure management system using:

  • 🎯 Argo CD for GitOps automation
  • ⚡ Crossplane for infrastructure provisioning
  • 🔄 App-of-Apps pattern for scalable application management
  • 📦 MetalLB as the infrastructure example
  • 🎭 Sync waves for dependency management

Result: Infrastructure changes now happen through Git commits, with full automation and zero manual intervention.

The Problem I Solved

Managing Kubernetes infrastructure traditionally sucks:

# The old way - manual and error-prone
kubectl apply -f metallb-config.yaml
kubectl apply -f ingress-controller.yaml
kubectl apply -f monitoring-stack.yaml
# Oh no! Order matters... 💥
# Which version is running in production? 🤷‍♂️
# Who made that change? 🕵️‍♂️
Enter fullscreen mode Exit fullscreen mode

I wanted infrastructure that:

  • ✅ Lives in Git (version controlled)
  • ✅ Deploys automatically (no manual steps)
  • ✅ Handles dependencies (no ordering issues)
  • ✅ Self-heals (drift detection & correction)
  • ✅ Provides audit trails (who, what, when)

The Solution Architecture 

graph LR
    A[Git Commit] --> B[Argo CD]
    B --> C[Crossplane]
    C --> D[Kubernetes Resources]
    B --> E[Sync Waves]
    E --> F[Ordered Deployment]
Enter fullscreen mode Exit fullscreen mode

🎯 Step 1: App-of-Apps Pattern

Instead of managing 50+ individual Argo CD applications, I use the App-of-Apps pattern:

# One app to rule them all
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: homelab-root
  annotations:
    argocd.argoproj.io/sync-wave: "-1"
spec:
  source:
    repoURL: https://github.com/jamilshaikh07/homelab-gitops.git
    path: apps  # 👈 All child apps live here
  syncPolicy:
    automated:
      prune: true      # 🗑️ Clean up deleted resources
      selfHeal: true   # 🔧 Fix manual changes
Enter fullscreen mode Exit fullscreen mode

Benefits:

  • One root app manages everything
  • New apps = just add YAML files
  • Automatic discovery and deployment

⚡ Step 2: Crossplane for Infrastructure

Crossplane lets me manage infrastructure through Kubernetes APIs. Here's the magic:

# Instead of direct kubectl apply...
apiVersion: kubernetes.crossplane.io/v1alpha1
kind: Object
metadata:
  name: metallb-ipaddresspool
  annotations:
    argocd.argoproj.io/sync-wave: "2"  # 👈 Depends on provider
spec:
  providerConfigRef:
    name: in-cluster
  forProvider:
    manifest:
      apiVersion: metallb.io/v1beta1
      kind: IPAddressPool
      metadata:
        name: homelab-pool
        namespace: metallb-system
      spec:
        addresses:
          - 10.20.0.81-10.20.0.99  # 🎯 My LoadBalancer IP range
Enter fullscreen mode Exit fullscreen mode

Why Crossplane Objects?

  • 🔄 Continuous reconciliation (drift detection)
  • 📊 Rich status reporting (health, errors)
  • 🎭 Dependency management (waits for providers)
  • 🔒 RBAC integration (secure access)

🎭 Step 3: Sync Waves for Dependencies

Order matters in infrastructure! I use sync waves to ensure proper sequencing:

# Wave -1: Root app-of-apps
argocd.argoproj.io/sync-wave: "-1"

# Wave 0: Install Crossplane providers
argocd.argoproj.io/sync-wave: "0"

# Wave 1: Provider configs + RBAC
argocd.argoproj.io/sync-wave: "1"

# Wave 2: Infrastructure resources
argocd.argoproj.io/sync-wave: "2"
Enter fullscreen mode Exit fullscreen mode

Result: No more "CRD not found" or "provider not ready" errors! 🎉

🔐 Step 4: RBAC - The Critical Missing Piece

Crossplane needs permissions to manage your infrastructure. This is often overlooked:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: crossplane-provider-kubernetes-metallb
rules:
- apiGroups: ["metallb.io"]
  resources: ["ipaddresspools", "l2advertisements"]
  verbs: ["*"]
---
# Bind to the provider's service account
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: crossplane-provider-kubernetes-metallb
roleRef:
  kind: ClusterRole
  name: crossplane-provider-kubernetes-metallb
subjects:
- kind: ServiceAccount
  name: provider-kubernetes-xxxxx  # 👈 Get from kubectl
  namespace: crossplane-system
Enter fullscreen mode Exit fullscreen mode

Pro tip: Restart provider pods after RBAC changes!

🧪 Testing the Complete Pipeline

Time for the moment of truth:

# 1. Bootstrap (one-time manual step)
kubectl apply -f argocd/app-of-apps.yaml

# 2. Check GitOps automation
kubectl -n argocd get applications
NAME                             SYNC STATUS   HEALTH STATUS
crossplane-provider-kubernetes   Synced        Healthy ✅
homelab-root                     Synced        Healthy ✅
metallb-config                   Synced        Healthy ✅

# 3. Verify infrastructure was created
kubectl -n metallb-system get ipaddresspools.metallb.io
NAME           ADDRESSES
homelab-pool   ["10.20.0.81-10.20.0.99"] ✅

# 4. Test LoadBalancer functionality
kubectl create deployment nginx --image=nginx
kubectl expose deployment nginx --type=LoadBalancer --port=80

kubectl get svc nginx
NAME    TYPE           EXTERNAL-IP   PORT(S)
nginx   LoadBalancer   10.20.0.82    80:30114/TCP ✅

# 5. Verify connectivity
curl http://10.20.0.82
<!DOCTYPE html>
<html>
<head><title>Welcome to nginx!</title></head>
# 🎉 SUCCESS!
Enter fullscreen mode Exit fullscreen mode

It works! Infrastructure deployed entirely through GitOps! 🚀

📁 Repository Structure 

homelab-gitops/
├── argocd/
│   └── app-of-apps.yaml              # 🏠 Root application
├── apps/
│   ├── crossplane-provider-kubernetes-app.yaml
│   └── metallb-config-app.yaml       # 👶 Child applications
├── crossplane/
│   └── provider-kubernetes/
│       ├── provider.yaml             # ⚡ Crossplane provider
│       ├── providerconfig.yaml       # ⚙️ Configuration
│       └── rbac.yaml                 # 🔐 Permissions
└── metallb/
    ├── metallb-ipaddresspool.yaml    # 🌐 Infrastructure
    └── metallb-l2advertisement.yaml  # 📦 Resources
Enter fullscreen mode Exit fullscreen mode

💡 Key Lessons Learned

1. API Versions Are Critical

Different provider versions use different APIs:

# v0.13.0 uses v1alpha1
apiVersion: kubernetes.crossplane.io/v1alpha1

# Newer versions use v1alpha2  
apiVersion: kubernetes.crossplane.io/v1alpha2
Enter fullscreen mode Exit fullscreen mode

2. Bootstrap is Still Manual

Even with full GitOps, you need one manual step:

kubectl apply -f argocd/app-of-apps.yaml
Enter fullscreen mode Exit fullscreen mode

After this, everything else is automated!

3. RBAC Debugging

If Crossplane objects stay "NotReady":

# Check provider permissions
kubectl -n crossplane-system describe object metallb-ipaddresspool

# Common fix: restart provider after RBAC changes
kubectl -n crossplane-system delete pod -l pkg.crossplane.io/provider=provider-kubernetes
Enter fullscreen mode Exit fullscreen mode

4. YAML Formatting Matters

Watch your indentation! This kept my root app OutOfSync:

# ❌ Wrong
destination:
  server: https://kubernetes.default.svc
namespace: crossplane-system

# ✅ Correct  
destination:
  server: https://kubernetes.default.svc
  namespace: crossplane-system
Enter fullscreen mode Exit fullscreen mode

🚀 What's Next?

This foundation scales to manage ANY infrastructure:

# Database clusters
kind: PostgreSQLCluster

# Service meshes  
kind: Istio

# Monitoring stacks
kind: PrometheusStack

# Certificate management
kind: ClusterIssuer

# Storage solutions
kind: StorageClass
Enter fullscreen mode Exit fullscreen mode

The pattern stays the same:

  1. Define in YAML
  2. Commit to Git
  3. Argo CD syncs automatically
  4. Crossplane provisions infrastructure
  5. Profit! 💰

🎉 Results

Before:

  • Manual kubectl commands
  • Configuration drift
  • No audit trail
  • Deployment anxiety 😰

After:

  • Infrastructure as Code
  • Git-driven deployments
  • Automatic drift correction
  • Pull request reviews
  • Confidence in production 😎

Infrastructure changes are now as simple as creating a pull request!

🔗 Resources

What infrastructure will you GitOps next? Drop a comment and let me know what you're planning to automate! 👇

Follow me for more cloud-native and DevOps content! 🚀

Top comments (1)

Collapse
 
anik_sikder_313 profile image
Anik Sikder •

This is a textbook example of how GitOps can evolve from a deployment strategy into full-stack infrastructure orchestration. The use of Crossplane as a control plane abstraction paired with Argo CD’s sync waves is a brilliant way to tame dependency hell and enforce sequencing without brittle scripts.

The App-of-Apps pattern adds clarity and scalability, especially in environments where infra and app lifecycles diverge. And the RBAC integration with Crossplane? That’s the kind of operational hygiene that often gets overlooked until it breaks something critical.

Appreciate how you’ve framed this not just as tooling, but as a mindset shift from manual ops to declarative, auditable infrastructure. Posts like this help teams move from “GitOps curious” to production-ready.