DEV Community: Dinesh Reddy

Automating Kubernetes Sealed Secrets Management with Jenkins in a Multi-Cloud Environment -Part1

Dinesh Reddy — Wed, 05 Feb 2025 02:00:11 +0000

🚀 Automating Kubernetes Sealed Secrets Management with Jenkins in a Multi-Cloud Environment

🗒️ Introduction

Managing sensitive data like API keys, passwords, and certificates securely in Kubernetes can be challenging—especially in enterprise environments with multiple clusters. In this guide, we'll explore how to automate Kubernetes Sealed Secrets management using Jenkins, ensuring security, scalability, and efficiency across multi-cloud environments such as AKS, GKE, and EKS.

🛡️ Background

In enterprise environments, sensitive information like API keys, passwords, and certificates must be managed securely. Kubernetes, widely used for container orchestration, stores such data as secrets. However, plain-text Kubernetes secrets are not safe for version control systems or manual handling.

To address this, Sealed Secrets encrypt sensitive data so it can be safely stored and shared. Deploying and managing these Sealed Secrets efficiently in Amazon AKS requires automation to ensure security, scalability, and ease of use.

⚠️ The Challenge

Enterprises often face several challenges:

Security Risks: Storing secrets in plain text makes them vulnerable to unauthorized access.
Manual Errors: Manual handling increases the risk of mistakes.
Complex Management: Managing secrets across environments (dev, staging, prod) is time-consuming.
Lack of Automation: Kubernetes lacks built-in automation for secret encryption and deployment.

💡 The Solution: Jenkins + Docker-in-Docker (DIND)

A Jenkins pipeline, combined with a Docker-in-Docker (DIND) container, automates the secure management of Sealed Secrets in AKS clusters. Key features include:

🔐 Strong Encryption: Uses the Sealed Secrets Controller, ensuring only the AKS cluster can decrypt data.
📦 Centralized Management: Jenkins standardizes secret generation and deployment across environments.
🛡️ Security Compliance: Secrets are encrypted before storage to meet strict security standards.
🔑 RBAC Integration: Kubernetes Role-Based Access Control (RBAC) adds an extra layer of security.

🔗 Check the GitHub Repository for Reference

⚙️ Jenkins Pipeline Workflow

1️⃣ Prepare the Workspace

Set up the environment for processing secrets and generating artifacts:

mkdir -p /tmp/jenkins-k8s-apply
mkdir -p ${WORKSPACE}/sealed-secrets-artifacts
rm -f /tmp/jenkins-k8s-apply/* || true
ls -la /tmp/jenkins-k8s-apply || echo "Directory is empty"

Why? A clean workspace ensures no residual sensitive data remains. ✅

2️⃣ Process Base64 Encoded Secrets

Decode the provided secrets.yaml file:

echo ${SECRETS_YAML} > /tmp/jenkins-k8s-apply/secrets.yaml.b64
base64 --decode < /tmp/jenkins-k8s-apply/secrets.yaml.b64 > /tmp/jenkins-k8s-apply/secrets.yaml
ls -l /tmp/jenkins-k8s-apply/secrets.yaml
head -n 5 /tmp/jenkins-k8s-apply/secrets.yaml | grep -v 'data:' || echo "File appears to be empty"

3️⃣ Apply Kubernetes Config & Fetch Public Certificate

Fetch the Sealed Secrets Controller’s public certificate:

ls -l ${KUBECONFIG}  # Validate kubeconfig

docker run --rm \
  -v ${KUBECONFIG}:/tmp/kubeconfig \
  -v /tmp/jenkins-k8s-apply/secrets.yaml:/tmp/secrets.yaml \
  docker-dind-kube-secret kubeseal \
    --controller-name=sealed-secrets \
    --controller-namespace=kube-system \
    --kubeconfig=/tmp/kubeconfig \
    --fetch-cert > /tmp/jenkins-k8s-apply/sealed-secrets-cert.pem

ls -l /tmp/jenkins-k8s-apply/sealed-secrets-cert.pem  # Validate certificate

4️⃣ Create Sealed Secrets

Encrypt the secrets using the public certificate:

docker run --rm \
  -v ${KUBECONFIG}:/tmp/kubeconfig \
  -v /tmp/jenkins-k8s-apply/secrets.yaml:/tmp/secrets.yaml \
  -v /tmp/jenkins-k8s-apply/sealed-secrets-cert.pem:/tmp/sealed-secrets-cert.pem \
  docker-dind-kube-secret sh -c "kubeseal \
    --controller-name=sealed-secrets \
    --controller-namespace=kube-system \
    --format yaml \
    --cert /tmp/sealed-secrets-cert.pem \
    --namespace=${NAMESPACE} \
    < /tmp/secrets.yaml" > ${WORKSPACE}/sealed-secrets-artifacts/sealed-secrets.yaml

5️⃣ Generate Documentation

Create metadata to track secret deployments:

echo "Generated on: $(date)" > ${WORKSPACE}/sealed-secrets-artifacts/README.txt
echo "Namespace: ${NAMESPACE}" >> ${WORKSPACE}/sealed-secrets-artifacts/README.txt
echo "Controller: sealed-secrets" >> ${WORKSPACE}/sealed-secrets-artifacts/README.txt
echo "Controller Namespace: kube-system" >> ${WORKSPACE}/sealed-secrets-artifacts/README.txt

🎯 Key Benefits

🔒 Enhanced Security: Encryption before storage mitigates the risk of data leaks.
⚡ Automation: Reduces manual effort, ensuring consistent workflows.
🌍 Scalability: Easily handles multiple environments and clusters.
✅ Compliance: Simplifies regulatory audits with secure secret management.

Seamlessly integrates with existing CI/CD workflows and AWS services—perfect for scaling across clusters. 🚀

📦 Part 2: Automating Kubernetes Sealed Secrets in Multi-Cloud Environments

In Part 2, we’ll expand this approach to multi-cloud environments, covering:

AKS (Non-Production) ✅
GKE (Production Cluster 1) ☁️
EKS (Production Cluster 2) 🛡️

Stay tuned for advanced multi-cloud secret management techniques! 💼

Automating Kubernetes Sealed Secrets Management with Jenkins in a Multi-Cloud Environment-Part2

Dinesh Reddy — Wed, 05 Feb 2025 01:17:56 +0000

Automating Secure Kubernetes Sealed Secrets with Jenkins Pipeline in a Multi-Cloud Environment

Managing Kubernetes secrets securely across multiple environments and clusters can be challenging. This Jenkins pipeline simplifies and automates the secure management of Kubernetes sealed secrets across Azure Kubernetes Service (AKS), Google Kubernetes Engine (GKE), and Amazon Elastic Kubernetes Service (EKS). It ensures secure handling of sensitive information while optimizing efficiency through dynamic environment setups and parallel processing.

🚀 Key Features

1. Dynamic Cluster Selection

The pipeline dynamically determines target clusters based on the ENVIRONMENT parameter:

Non-Production: Targets the AKS cluster using the Stage credential.
Production: Targets both GKE (Production Cluster 1) and EKS (Production Cluster 2) using Production_1 and Production_2 credentials, respectively.

2. Parallel Processing for Efficiency

To speed up multi-cluster operations, the Process Clusters stage runs workflows in parallel:

Production: Processes GKE and EKS clusters simultaneously.
Non-Production: Processes only the AKS cluster.

3. Secure Sealed Secrets Workflow

Decodes the Base64-encoded Secrets.yaml file.
Fetches the public certificate from the Sealed Secrets controller.
Encrypts the secrets for the respective cluster and namespace.
Generates sealed-secrets.yaml artifacts for secure deployment.

4. Dynamic and Reusable Pipeline

The cluster list and credentials are dynamically configured, making the pipeline adaptable for additional clusters or environments with minimal changes.

5. Post-Build Artifact Management

Archives artifacts (sealed-secrets.yaml, README.txt) per cluster.
Makes them accessible through the Jenkins UI for easy retrieval.

🔄 Jenkins Pipeline Script Explained

Parameters

parameters {
    string(name: 'NAMESPACE', defaultValue: 'default', description: "'Kubernetes namespace for the sealed secret')"
    choice(name: 'ENVIRONMENT', choices: ['Non-Production', 'Production'], description: "'Select the target environment')"
    base64File(name: 'SECRETS_YAML', description: "'Upload Base64-encoded Secrets.yaml file')"
    booleanParam(name: 'STORE_CERT', defaultValue: true, description: "'Store the public certificate for future use')"
}

NAMESPACE: Target namespace in Kubernetes.
ENVIRONMENT: Determines the operational environment.
SECRETS_YAML: Base64-encoded YAML file with sensitive data.
STORE_CERT: Option to archive the public certificate for future use.

Environment Variables

environment {
    WORK_DIR = '/tmp/jenkins-k8s-apply'
    CONTROLLER_NAMESPACE = 'kube-system'
    CONTROLLER_NAME = 'sealed-secrets'
    CERT_FILE = 'sealed-secrets-cert.pem'
    DOCKER_IMAGE = 'docker-dind-kube-secret'
    ARTIFACTS_DIR = 'sealed-secrets-artifacts'
}

Defines key directories and configurations, including the workspace, Sealed Secrets controller details, and Docker image.

Environment Setup

stage('Environment Setup') {
    steps {
        script {
            def clusters = params.ENVIRONMENT == 'Production' ? [
                [id: 'prod-cluster-1', name: 'Production Cluster 1', credentialId: 'Production_1'],
                [id: 'prod-cluster-2', name: 'Production Cluster 2', credentialId: 'Production_2']
            ] : [
                [id: 'non-prod-cluster', name: 'Non-Production Cluster', credentialId: 'Stage']
            ];

            env.CLUSTER_IDS = clusters.collect { it.id }.join(',');
            clusters.each { cluster ->
                env["CLUSTER_${cluster.id}_NAME"] = cluster.name;
                env["CLUSTER_${cluster.id}_CRED"] = cluster.credentialId;
            }
        }
    }
}

Dynamically sets up clusters and credentials based on the environment.

Preparing the Workspace

stage('Prepare Workspace') {
    steps {
        script {
            sh """
                mkdir -p ${WORK_DIR}
                mkdir -p ${WORKSPACE}/${ARTIFACTS_DIR}
                rm -f ${WORK_DIR}/* || true
                rm -rf ${WORKSPACE}/${ARTIFACTS_DIR}/* || true
            """

            writeFile file: "${WORK_DIR}/secrets.yaml.b64", text: params.SECRETS_YAML;
            sh "base64 --decode < ${WORK_DIR}/secrets.yaml.b64 > ${WORK_DIR}/secrets.yaml";
        }
    }
}

Prepares directories, cleans up old artifacts, and decodes the Base64 secrets file.

Processing Clusters in Parallel

stage('Process Clusters') {
    steps {
        script {
            def parallelStages = [:];

            env.CLUSTER_IDS.split(',').each { clusterId ->
                def clusterName = env["CLUSTER_${clusterId}_NAME"];
                def credentialId = env["CLUSTER_${clusterId}_CRED"];

                parallelStages[clusterName] = {
                    stage("Process ${clusterName}") {
                        withCredentials([file(credentialsId: credentialId, variable: 'KUBECONFIG')]) {
                            sh "docker run --rm -v \${KUBECONFIG}:/tmp/kubeconfig -v ${WORK_DIR}/secrets.yaml:/tmp/secrets.yaml ${DOCKER_IMAGE} kubeseal --controller-name=${CONTROLLER_NAME} --controller-namespace=${CONTROLLER_NAMESPACE} --cert /tmp/sealed-secrets-cert.pem --namespace=${params.NAMESPACE} < /tmp/secrets.yaml > ${WORKSPACE}/${ARTIFACTS_DIR}/${clusterId}/sealed-secrets.yaml";
                        }
                    }
                }
            }

            parallel parallelStages;
        }
    }
}

Processes each cluster in parallel to optimize execution time.

Post-Build Actions

post {
    always {
        sh "rm -rf ${WORK_DIR}";
        archiveArtifacts artifacts: "${ARTIFACTS_DIR}/*/**", fingerprint: true;
    }
    success {
        echo "Pipeline completed successfully!";
    }
    failure {
        echo "Pipeline failed. Check logs for details.";
    }
}

Cleans up the workspace.
Archives artifacts for easy retrieval.

🌟 Key Benefits

Dynamic Configuration: Easy to add new clusters without major code changes.
Parallel Processing: Optimized runtime for multi-cluster environments.
Secure Operations: End-to-end encryption with Kubernetes Sealed Secrets.
Multi-Cloud Ready: Works seamlessly with AKS, GKE, and EKS.

This Jenkins pipeline provides a robust, secure, and efficient way to manage Kubernetes secrets across multiple clouds. Feel free to adapt it to your organization's needs! 🚀