DEV Community

giveitatry
giveitatry

Posted on

How I Learned to Redeploy Applications on k3s with GitLab CI/CD – Step by Step

#kubernetes #tutorial #devops #cicd

Automating deployments is a huge time saver, but when I first tried it with my k3s cluster, I ran into a few obstacles. I wanted GitLab CI to redeploy my app automatically after every successful push, but I didn’t want to give the pipeline full cluster access—just enough to restart deployments. Here’s how I figured it out.

Step 1: Understanding Permissions and Creating a Service Account

I started by realizing that GitLab CI needs credentials to talk to Kubernetes. Instead of giving it cluster admin rights (which felt unsafe), I created a service account limited to my application namespace.

kubectl create namespace my-namespace
kubectl create serviceaccount gitlab-deployer -n my-namespace
Enter fullscreen mode Exit fullscreen mode

At this point, the service account exists, but it doesn’t know what it’s allowed to do. That comes next.

Step 2: Defining What the CI Can Do With a Role

I wanted the CI to redeploy apps without touching anything else. I created a Role giving permission only to deployments and pods in my namespace:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: gitlab-deployer-role
  namespace: my-namespace
rules:
  - apiGroups: ["apps"]
    resources: ["deployments"]
    verbs: ["get", "list", "watch", "update", "patch"]
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get", "list", "watch"]
Enter fullscreen mode Exit fullscreen mode

Applied it:

kubectl apply -f gitlab-deployer-role.yaml
Enter fullscreen mode Exit fullscreen mode

Step 3: Binding the Role to the Service Account

The service account needs to know it can use the role. I created a RoleBinding:

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: gitlab-deployer-binding
  namespace: my-namespace
subjects:
  - kind: ServiceAccount
    name: gitlab-deployer
    namespace: my-namespace
roleRef:
  kind: Role
  name: gitlab-deployer-role
  apiGroup: rbac.authorization.k8s.io
Enter fullscreen mode Exit fullscreen mode

Applied it:

kubectl apply -f gitlab-deployer-rolebinding.yaml
Enter fullscreen mode Exit fullscreen mode

Now, GitLab CI can safely interact with deployments in my-namespace and nothing else.

Step 4: Generating a Token for GitLab CI

I needed a token so GitLab CI could authenticate as the service account. With k3s (and Kubernetes 1.24+), I learned that secrets are no longer created automatically, so the simplest approach is:

kubectl create token gitlab-deployer -n my-namespace
Enter fullscreen mode Exit fullscreen mode

[SEE comments]

  • I copied this token and would later add it as a GitLab CI/CD variable called KUBE_TOKEN.
  • This way, the pipeline can log in to the cluster without exposing credentials in code.

Step 5: Finding the Kubernetes API URL

GitLab CI also needs the cluster’s API endpoint. I found it using:

kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}'
Enter fullscreen mode Exit fullscreen mode
  • Example: https://<k3s-server-ip>:6443
  • This would later go into a GitLab CI/CD variable called KUBE_API_URL.

Step 6: Getting the Kubernetes CA Certificate

I realized that CI also needs to verify the cluster securely. That’s where the CA certificate comes in.

There are two ways to get it:

Option 1: From kubeconfig

kubectl config view --raw --minify --flatten -o jsonpath='{.clusters[0].cluster.certificate-authority-data}' | base64 --decode
Enter fullscreen mode Exit fullscreen mode

Option 2: From k3s installation

cat /etc/rancher/k3s/ca.crt
Enter fullscreen mode Exit fullscreen mode

I copied the certificate including the BEGIN/END lines. Later in GitLab, I added it as a variable called KUBE_CA_PEM. This way, the CI pipeline could securely talk to the cluster over HTTPS.

Step 7: Adding Variables to GitLab CI/CD

At this point, I had three things the CI needed:

Variable Purpose
KUBE_TOKEN Authenticates as the service account
KUBE_API_URL Knows which cluster to talk to
KUBE_CA_PEM Verifies the cluster’s identity securely

I went to GitLab → Settings → CI/CD → Variables and added each one. I masked and protected them, so they wouldn’t appear in logs and would only be used on main branch.

Step 8: Writing the CI Job

This part took some trial and error. Initially, I used bitnami/kubectl, but got the error:

unknown command "sh" for "kubectl"
Enter fullscreen mode Exit fullscreen mode

I learned that minimal kubectl images sometimes don’t include a proper shell. Switching to ubuntu:22.04 and installing kubectl dynamically solved it.

Here’s my final working job:

stages:
  - deploy

deploy:
  stage: deploy
  image: ubuntu:22.04
  before_script:
    - apt-get update && apt-get install -y curl
    - curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
    - install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl
  script:
    # Save CA certificate from variable
    - echo "$KUBE_CA_PEM" > ca.crt

    # Configure kubectl with token and cluster info
    - kubectl config set-cluster k3s --server=$KUBE_API_URL --certificate-authority=ca.crt
    - kubectl config set-credentials gitlab --token=$KUBE_TOKEN
    - kubectl config set-context gitlab --cluster=k3s --user=gitlab --namespace=my-namespace
    - kubectl config use-context gitlab

    # Redeploy application
    - kubectl rollout restart deployment/my-app
    - kubectl rollout status deployment/my-app -n my-namespace
  only:
    - main
Enter fullscreen mode Exit fullscreen mode

Notice how the variables we set earlier (KUBE_CA_PEM, KUBE_TOKEN, KUBE_API_URL) are plugged directly into the CI job, making the process secure and reusable.

Step 9: Optional Improvements I Learned

  • Caching kubectl to avoid downloading every CI run: 
cache:
  paths:
    - /usr/local/bin/kubectl
Enter fullscreen mode Exit fullscreen mode
  • Parameterized deployments for multiple apps: 
- kubectl rollout restart deployment/$DEPLOYMENT_NAME -n $NAMESPACE
Enter fullscreen mode Exit fullscreen mode
  • Rollout status check ensures the CI waits until pods are ready.

Step 10: Troubleshooting Tips

Symptom Cause Solution
unknown command "sh" for "kubectl" Minimal kubectl image lacks shell Use ubuntu:22.04 or lachlanevenson/k8s-kubectl
Secrets for service account empty Kubernetes 1.24+ doesn’t auto-create SA secrets Use kubectl create token gitlab-deployer -n my-namespace
TLS verification fails CA certificate missing Add KUBE_CA_PEM variable from kubeconfig or /etc/rancher/k3s/ca.crt
Permission denied Role/RoleBinding not applied Verify kubectl auth can-i rollout restart deployment/my-app -n my-namespace --as=system:serviceaccount:my-namespace:gitlab-deployer

Step 11: Verifying Everything Works

From a local machine or runner:

kubectl --token=<TOKEN> --server=<KUBE_API_URL> --certificate-authority=ca.crt get deployments -n my-namespace
Enter fullscreen mode Exit fullscreen mode

You should only see deployments in your namespace—no access to other namespaces or cluster-wide resources.

Result and Takeaways

After some trial and error, I ended up with a secure, automated deployment pipeline that:

  • Uses minimal permissions for safety
  • Handles TLS verification securely
  • Works reliably with k3s and GitLab CI/CD
  • Automatically redeploys apps after a successful push

Top comments (1)

Collapse
 
giveitatry profile image
giveitatry • • Edited

Note that: kubectl create token gitlab-deployer -n my-namespace
will create temporary token. It may case that you pipeline stops working after ~1h.

To make it stable, you can manually create a ServiceAccount token Secret that does not expire.

1. Create a Secret

Run:

kubectl apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: gitlab-deployer-token
  namespace: mtdb
  annotations:
    kubernetes.io/service-account.name: gitlab-deployer
type: kubernetes.io/service-account-token
EOF
Enter fullscreen mode Exit fullscreen mode

2. Wait a few seconds, then get the token: 

kubectl get secret gitlab-deployer-token -n mtdb -o jsonpath='{.data.token}' | base64 --decode
Enter fullscreen mode Exit fullscreen mode

You’ll get a long-lived static token.
That’s the one you should put in your GitLab CI/CD variable KUBE_TOKEN.

This token will not change or expire, unless:

  • You delete the Secret, or
  • You delete the ServiceAccount or namespace.

Security Tip

This static token gives long-term access to your cluster, so:

  • Keep it namespace-limited (as we did earlier).
  • Mask and protect it in GitLab CI variables.
  • Optionally, use Kubernetes RBAC to make it read/write only on specific resources.