DEV Community

Aisalkyn Aidarova
Aisalkyn Aidarova

Posted on

production statefulset argocd

#aws #devops #infrastructure #kubernetes 
GitHub
  ↓
Argo CD
  ↓
Kubernetes
  ↓
StatefulSet
  ├── Headless Service
  ├── ConfigMap
  ├── ServiceAccount
  ├── StatefulSet
  ├── PVC per Pod
  ├── PDB
  └── StorageClass
        ↓
      AWS EBS
Enter fullscreen mode Exit fullscreen mode

Repository students should create 

restaurant-statefulset-gitops/
│
├── README.md
├── argocd/
│   └── application.yaml
│
└── k8s/
    ├── namespace.yaml
    ├── serviceaccount.yaml
    ├── configmap.yaml
    ├── storageclass.yaml
    ├── headless-service.yaml
    ├── service.yaml
    ├── statefulset.yaml
    ├── pdb.yaml
    └── kustomization.yaml
Enter fullscreen mode Exit fullscreen mode

k8s/namespace.yaml 

apiVersion: v1
kind: Namespace
metadata:
  name: restaurant
Enter fullscreen mode Exit fullscreen mode

k8s/serviceaccount.yaml 

apiVersion: v1
kind: ServiceAccount
metadata:
  name: restaurant
  namespace: restaurant
automountServiceAccountToken: false
Enter fullscreen mode Exit fullscreen mode

k8s/configmap.yaml 

apiVersion: v1
kind: ConfigMap
metadata:
  name: restaurant-config
  namespace: restaurant
data:
  APP_ENV: "production"
  APP_NAME: "restaurant-company"
Enter fullscreen mode Exit fullscreen mode

k8s/storageclass.yaml

For EKS production, use the EBS CSI provisioner:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: restaurant-gp3
provisioner: ebs.csi.aws.com
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
parameters:
  type: gp3
  encrypted: "true"
Enter fullscreen mode Exit fullscreen mode

Explain:

PVC
 ↓
StorageClass
 ↓
EBS CSI Driver
 ↓
AWS creates EBS volume
 ↓
PV
Enter fullscreen mode Exit fullscreen mode

Your students must verify:

kubectl get csidriver
Enter fullscreen mode Exit fullscreen mode

They need:

ebs.csi.aws.com
Enter fullscreen mode Exit fullscreen mode

before this storage can work.

k8s/headless-service.yaml

StatefulSet uses this for stable network identity:

apiVersion: v1
kind: Service
metadata:
  name: restaurant-headless
  namespace: restaurant
spec:
  clusterIP: None

  selector:
    app: restaurant

  ports:
    - name: http
      port: 80
      targetPort: http
Enter fullscreen mode Exit fullscreen mode

Now you can teach:

restaurant-0.restaurant-headless.restaurant.svc.cluster.local
restaurant-1.restaurant-headless.restaurant.svc.cluster.local
restaurant-2.restaurant-headless.restaurant.svc.cluster.local
Enter fullscreen mode Exit fullscreen mode

k8s/service.yaml

This is the normal Service clients use:

apiVersion: v1
kind: Service
metadata:
  name: restaurant
  namespace: restaurant
spec:
  type: ClusterIP

  selector:
    app: restaurant

  ports:
    - name: http
      port: 80
      targetPort: http
Enter fullscreen mode Exit fullscreen mode

k8s/statefulset.yaml

Using the ECR image you've been running:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: restaurant
  namespace: restaurant

spec:
  serviceName: restaurant-headless

  replicas: 3

  podManagementPolicy: OrderedReady

  updateStrategy:
    type: RollingUpdate

  selector:
    matchLabels:
      app: restaurant

  template:
    metadata:
      labels:
        app: restaurant

    spec:
      serviceAccountName: restaurant

      terminationGracePeriodSeconds: 30

      securityContext:
        fsGroup: 101

      containers:
        - name: restaurant

          image: 584612873567.dkr.ecr.us-east-1.amazonaws.com/resto-comp:latest

          imagePullPolicy: Always

          ports:
            - name: http
              containerPort: 80

          envFrom:
            - configMapRef:
                name: restaurant-config

          resources:
            requests:
              cpu: 100m
              memory: 128Mi

            limits:
              cpu: 500m
              memory: 512Mi

          readinessProbe:
            httpGet:
              path: /
              port: http
            initialDelaySeconds: 5
            periodSeconds: 10
            timeoutSeconds: 3
            failureThreshold: 3

          livenessProbe:
            httpGet:
              path: /
              port: http
            initialDelaySeconds: 15
            periodSeconds: 20
            timeoutSeconds: 3
            failureThreshold: 3

          volumeMounts:
            - name: restaurant-data
              mountPath: /data

  volumeClaimTemplates:
    - metadata:
        name: restaurant-data

      spec:
        accessModes:
          - ReadWriteOnce

        storageClassName: restaurant-gp3

        resources:
          requests:
            storage: 5Gi
Enter fullscreen mode Exit fullscreen mode

This creates:

restaurant-0
   ↓
restaurant-data-restaurant-0
   ↓
EBS-0


restaurant-1
   ↓
restaurant-data-restaurant-1
   ↓
EBS-1


restaurant-2
   ↓
restaurant-data-restaurant-2
   ↓
EBS-2
Enter fullscreen mode Exit fullscreen mode

k8s/pdb.yaml 

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: restaurant
  namespace: restaurant

spec:
  minAvailable: 2

  selector:
    matchLabels:
      app: restaurant
Enter fullscreen mode Exit fullscreen mode

Explain that PDB is about voluntary disruptions, such as node maintenance/draining. It doesn't guarantee that two Pods can never fail.

k8s/kustomization.yaml 

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - namespace.yaml
  - serviceaccount.yaml
  - configmap.yaml
  - storageclass.yaml
  - headless-service.yaml
  - service.yaml
  - statefulset.yaml
  - pdb.yaml
Enter fullscreen mode Exit fullscreen mode

Students can test the complete manifest before Argo CD:

kubectl kustomize k8s/
Enter fullscreen mode Exit fullscreen mode

Then:

kubectl apply --dry-run=server -k k8s/
Enter fullscreen mode Exit fullscreen mode

Argo CD Application

Now create:

argocd/application.yaml
Enter fullscreen mode Exit fullscreen mode

Use:

apiVersion: argoproj.io/v1alpha1
kind: Application

metadata:
  name: restaurant-statefulset
  namespace: argocd

spec:
  project: default

  source:
    repoURL: https://github.com/jumptotechschooldevops/restaurant-statefulset-gitops.git
    targetRevision: main
    path: k8s

  destination:
    server: https://kubernetes.default.svc
    namespace: restaurant

  syncPolicy:
    automated:
      prune: true
      selfHeal: true

    syncOptions:
      - CreateNamespace=true
      - PruneLast=true
Enter fullscreen mode Exit fullscreen mode

Students need to replace repoURL if they're using their own repository.

Now teach the GitOps flow:

Developer
   ↓
git push

GitHub
   ↓
Argo CD watches repository
   ↓
desired state
   ↓
Kubernetes
   ↓
StatefulSet
   ↓
Pods + PVCs
Enter fullscreen mode Exit fullscreen mode

Apply the Argo application:

kubectl apply -f argocd/application.yaml
Enter fullscreen mode Exit fullscreen mode

Then:

kubectl get applications -n argocd
Enter fullscreen mode Exit fullscreen mode

Watch:

kubectl get pods -n restaurant -w
Enter fullscreen mode Exit fullscreen mode

Then:

kubectl get statefulset -n restaurant
Enter fullscreen mode Exit fullscreen mode 
kubectl get pvc -n restaurant
Enter fullscreen mode Exit fullscreen mode 
kubectl get pv
Enter fullscreen mode Exit fullscreen mode 
kubectl get service -n restaurant
Enter fullscreen mode Exit fullscreen mode

Student StatefulSet test

Once everything is Running and PVCs are Bound:

kubectl exec -it restaurant-0 -n restaurant -- sh
Enter fullscreen mode Exit fullscreen mode

Inside:

echo "DATA FROM RESTAURANT-0" > /data/student.txt
cat /data/student.txt
exit
Enter fullscreen mode Exit fullscreen mode

Now:

kubectl delete pod restaurant-0 -n restaurant
Enter fullscreen mode Exit fullscreen mode

Watch:

kubectl get pods -n restaurant -w
Enter fullscreen mode Exit fullscreen mode

Once restaurant-0 is back:

kubectl exec restaurant-0 -n restaurant -- \
  cat /data/student.txt
Enter fullscreen mode Exit fullscreen mode

Expected:

DATA FROM RESTAURANT-0
Enter fullscreen mode Exit fullscreen mode

Then compare Pod 1:

kubectl exec restaurant-1 -n restaurant -- \
  cat /data/student.txt
Enter fullscreen mode Exit fullscreen mode

It should not contain Pod 0's file because Pod 1 has its own PVC.

Argo CD self-healing lab

This is the part I'd definitely include.

Repository says:

replicas: 3
Enter fullscreen mode Exit fullscreen mode

Student manually changes Kubernetes:

kubectl scale statefulset restaurant \
  --replicas=1 \
  -n restaurant
Enter fullscreen mode Exit fullscreen mode

Then:

kubectl get statefulset -n restaurant
Enter fullscreen mode Exit fullscreen mode

Ask:

Git says 3. Cluster says 1. Which one wins?

Because you enabled:

automated:
  selfHeal: true
Enter fullscreen mode Exit fullscreen mode

Argo CD should reconcile the cluster back toward the Git desired state.

Teach:

Git = desired state

Cluster = actual state

       ↓

Argo CD compares them

Desired != Actual
       ↓
Argo CD reconciles
       ↓
Actual → Desired
Enter fullscreen mode Exit fullscreen mode

Git change lab

Now tell students don't use kubectl to make the desired change.

Change:

replicas: 3
Enter fullscreen mode Exit fullscreen mode

to:

replicas: 2
Enter fullscreen mode Exit fullscreen mode

Then:

git add .
git commit -m "Scale restaurant StatefulSet to 2 replicas"
git push
Enter fullscreen mode Exit fullscreen mode

Argo CD:

Git changed
   ↓
Argo detects change
   ↓
Sync
   ↓
StatefulSet changes
Enter fullscreen mode Exit fullscreen mode

Check:

kubectl get pods -n restaurant
Enter fullscreen mode Exit fullscreen mode

Production lesson: never put passwords in Git

Do not teach students to commit this:

DB_PASSWORD: mypassword123
Enter fullscreen mode Exit fullscreen mode

into ConfigMap or a plaintext Secret in a public repository.

Explain:

ConfigMap
   ↓
non-sensitive configuration

APP_ENV
APP_NAME
LOG_LEVEL
Enter fullscreen mode Exit fullscreen mode

Secrets should come from an appropriate secret-management workflow, such as AWS Secrets Manager with an external-secrets integration, rather than plaintext Git.

Very important architecture note for students

Your Restaurant Company application is a frontend served by Nginx.

In a real production architecture, I would teach them:

Frontend
Restaurant Company
      ↓
Deployment
      ↓
3 interchangeable Pods


Database
PostgreSQL/MySQL/etc.
      ↓
Stateful workload
      ↓
stable identity
      +
persistent storage
Enter fullscreen mode Exit fullscreen mode

So tell them:

Мы запускаем Restaurant Company как StatefulSet для лаборатории, чтобы увидеть identity, PVC, PV, EBS и Argo CD.

Это не означает, что каждый frontend должен быть StatefulSet.

That's a very important production distinction.

Final student architecture 

                    GITHUB
                       │
                       │ git push
                       ▼
                    ARGO CD
                       │
                  sync/self-heal
                       │
                       ▼
                EKS KUBERNETES
                       │
                 StatefulSet
                       │
       ┌───────────────┼───────────────┐
       ▼               ▼               ▼
 restaurant-0     restaurant-1     restaurant-2
       │               │               │
       ▼               ▼               ▼
    PVC-0            PVC-1           PVC-2
       │               │               │
       ▼               ▼               ▼
    EBS-0            EBS-1           EBS-2

           Headless Service
                  │
           Stable DNS Identity
Enter fullscreen mode Exit fullscreen mode

Top comments (0)