Automating Kubernetes Sealed Secrets Management with Jenkins in a Multi-Cloud Environment-Part2

Automating Secure Kubernetes Sealed Secrets with Jenkins Pipeline in a Multi-Cloud Environment

Managing Kubernetes secrets securely across multiple environments and clusters can be challenging. This Jenkins pipeline simplifies and automates the secure management of Kubernetes sealed secrets across Azure Kubernetes Service (AKS), Google Kubernetes Engine (GKE), and Amazon Elastic Kubernetes Service (EKS). It ensures secure handling of sensitive information while optimizing efficiency through dynamic environment setups and parallel processing.

🚀 Key Features

1. Dynamic Cluster Selection

The pipeline dynamically determines target clusters based on the ENVIRONMENT parameter:

• Non-Production: Targets the AKS cluster using the Stage credential.
• Production: Targets both GKE (Production Cluster 1) and EKS (Production Cluster 2) using Production_1 and Production_2 credentials, respectively.

2. Parallel Processing for Efficiency

To speed up multi-cluster operations, the Process Clusters stage runs workflows in parallel:

• Production: Processes GKE and EKS clusters simultaneously.
• Non-Production: Processes only the AKS cluster.

3. Secure Sealed Secrets Workflow

• Decodes the Base64-encoded Secrets.yaml file.
• Fetches the public certificate from the Sealed Secrets controller.
• Encrypts the secrets for the respective cluster and namespace.
• Generates sealed-secrets.yaml artifacts for secure deployment.

4. Dynamic and Reusable Pipeline

The cluster list and credentials are dynamically configured, making the pipeline adaptable for additional clusters or environments with minimal changes.

5. Post-Build Artifact Management

• Archives artifacts (sealed-secrets.yaml, README.txt) per cluster.
• Makes them accessible through the Jenkins UI for easy retrieval.

---

🔄 Jenkins Pipeline Script Explained

Parameters

parameters {
    string(name: 'NAMESPACE', defaultValue: 'default', description: "'Kubernetes namespace for the sealed secret')"
    choice(name: 'ENVIRONMENT', choices: ['Non-Production', 'Production'], description: "'Select the target environment')"
    base64File(name: 'SECRETS_YAML', description: "'Upload Base64-encoded Secrets.yaml file')"
    booleanParam(name: 'STORE_CERT', defaultValue: true, description: "'Store the public certificate for future use')"
}

• NAMESPACE: Target namespace in Kubernetes.
• ENVIRONMENT: Determines the operational environment.
• SECRETS_YAML: Base64-encoded YAML file with sensitive data.
• STORE_CERT: Option to archive the public certificate for future use.

Environment Variables

environment {
    WORK_DIR = '/tmp/jenkins-k8s-apply'
    CONTROLLER_NAMESPACE = 'kube-system'
    CONTROLLER_NAME = 'sealed-secrets'
    CERT_FILE = 'sealed-secrets-cert.pem'
    DOCKER_IMAGE = 'docker-dind-kube-secret'
    ARTIFACTS_DIR = 'sealed-secrets-artifacts'
}

Defines key directories and configurations, including the workspace, Sealed Secrets controller details, and Docker image.

Environment Setup

stage('Environment Setup') {
    steps {
        script {
            def clusters = params.ENVIRONMENT == 'Production' ? [
                [id: 'prod-cluster-1', name: 'Production Cluster 1', credentialId: 'Production_1'],
                [id: 'prod-cluster-2', name: 'Production Cluster 2', credentialId: 'Production_2']
            ] : [
                [id: 'non-prod-cluster', name: 'Non-Production Cluster', credentialId: 'Stage']
            ];

            env.CLUSTER_IDS = clusters.collect { it.id }.join(',');
            clusters.each { cluster ->
                env["CLUSTER_${cluster.id}_NAME"] = cluster.name;
                env["CLUSTER_${cluster.id}_CRED"] = cluster.credentialId;
            }
        }
    }
}

Dynamically sets up clusters and credentials based on the environment.

Preparing the Workspace

stage('Prepare Workspace') {
    steps {
        script {
            sh """
                mkdir -p ${WORK_DIR}
                mkdir -p ${WORKSPACE}/${ARTIFACTS_DIR}
                rm -f ${WORK_DIR}/* || true
                rm -rf ${WORKSPACE}/${ARTIFACTS_DIR}/* || true
            """

            writeFile file: "${WORK_DIR}/secrets.yaml.b64", text: params.SECRETS_YAML;
            sh "base64 --decode < ${WORK_DIR}/secrets.yaml.b64 > ${WORK_DIR}/secrets.yaml";
        }
    }
}

Prepares directories, cleans up old artifacts, and decodes the Base64 secrets file.

Processing Clusters in Parallel

stage('Process Clusters') {
    steps {
        script {
            def parallelStages = [:];

            env.CLUSTER_IDS.split(',').each { clusterId ->
                def clusterName = env["CLUSTER_${clusterId}_NAME"];
                def credentialId = env["CLUSTER_${clusterId}_CRED"];

                parallelStages[clusterName] = {
                    stage("Process ${clusterName}") {
                        withCredentials([file(credentialsId: credentialId, variable: 'KUBECONFIG')]) {
                            sh "docker run --rm -v \${KUBECONFIG}:/tmp/kubeconfig -v ${WORK_DIR}/secrets.yaml:/tmp/secrets.yaml ${DOCKER_IMAGE} kubeseal --controller-name=${CONTROLLER_NAME} --controller-namespace=${CONTROLLER_NAMESPACE} --cert /tmp/sealed-secrets-cert.pem --namespace=${params.NAMESPACE} < /tmp/secrets.yaml > ${WORKSPACE}/${ARTIFACTS_DIR}/${clusterId}/sealed-secrets.yaml";
                        }
                    }
                }
            }

            parallel parallelStages;
        }
    }
}

Processes each cluster in parallel to optimize execution time.

Post-Build Actions

post {
    always {
        sh "rm -rf ${WORK_DIR}";
        archiveArtifacts artifacts: "${ARTIFACTS_DIR}/*/**", fingerprint: true;
    }
    success {
        echo "Pipeline completed successfully!";
    }
    failure {
        echo "Pipeline failed. Check logs for details.";
    }
}

• Cleans up the workspace.
• Archives artifacts for easy retrieval.

---

🌟 Key Benefits

• Dynamic Configuration: Easy to add new clusters without major code changes.
• Parallel Processing: Optimized runtime for multi-cluster environments.
• Secure Operations: End-to-end encryption with Kubernetes Sealed Secrets.
• Multi-Cloud Ready: Works seamlessly with AKS, GKE, and EKS.

This Jenkins pipeline provides a robust, secure, and efficient way to manage Kubernetes secrets across multiple clouds. Feel free to adapt it to your organization's needs! 🚀