EKS Secret Management — with Golang, AWS ParameterStore and Terraform

Table of Contents

1. Introduction
2. InitContainer with GO binary
3. OIDC Federated Access for EKS Pods
4. Farewell

Introduction

Hey Folks! In this article, we are going to delve into a robust approach to Kubernetes secret management by utilizing the efficiency of Golang, the security and flexibility of AWS ParameterStore, the authentication power of OIDC, and the infrastructure-as-code advantages of Terraform. We will explore ways to enhance your cloud-based applications and significantly bolster your security posture, providing you with a comprehensive understanding of this innovative strategy for revolutionizing your secret management processes.

Keep in mind that we have a few prerequisites. To fully engage with the material and examples we provide, you’ll need an AWS Account, an EKS Cluster, and a configured Terraform project with the AWS provider.

SSM Parameters

To kick things off, let’s explore how we can manage secrets using AWS Systems Manager (SSM) Parameter Store. This service from AWS provides secure, hierarchical storage for configuration data management and secrets. Leveraging the Parameter Store can significantly enhance the security posture of your applications by segregating and securing sensitive information like database passwords, license codes, and API keys.

Let’s consider a Terraform script to create these SSM parameters, starting with a locals block. This block includes the projects in which we want to manage the secrets and the keys that need to be stored securely.

locals {
  projects = {
    demo-project = {
      team_id                 = "demo_team"
      namespace               = "demo"
      platform                = "fargate"
      fqdn                    = "www.huseynov.net"
      ssm_secret_keys         = ["AMQ_USER", "AMQ_PASS"]
      deployment_grace_period = 60
      vpa_update_mode         = "Initial"
      svc_config              = {
        service_port   = 8080
        target_port    = 80
        type           = "NodePort"
        lb_access_mode = "internet-facing"
        alb_group      = "demo"
      }
      hpa_config = {
        min           = 1
        max           = 3
        mem_threshold = 80
        cpu_threshold = 60
      }
    }
  }

  ssm_secret_keys = { for v in flatten([
    for project_name, parameters in local.projects : [
      for key in try(parameters.ssm_secret_keys, []) : {
        key          = key,
        namespace    = parameters.namespace,
        project_name = project_name,
        team_id      = try(parameters.team_id, var.cluster_namespace)
      }
    ]
    ]) : "${v.namespace}.${v.project_name}.${v.key}" => v
  }
}


resource "aws_ssm_parameter" "project_ssm_secrets" {
  for_each = local.ssm_secret_keys
  name     = "/eks/${var.cluster_name}/${each.value.namespace}/${each.value.project_name}/${each.value.key}"
  type     = "SecureString"
  value    = "placeholder"
  tags = merge(local.default_tags, {
    "eks.namespace" = each.value.namespace
    "eks.project"   = each.value.project_name
    "eks.team_id"   = each.value.team_id
    }
  )
  lifecycle {
    ignore_changes  = [value]
    prevent_destroy = true
  }
}

Please note that in our locals block, several parameters are of particular importance for this tutorial:

1. ssm_secret_keys: This specifies the secrets that we need to securely manage. These are essentially the names of the secrets we are storing in the AWS ParameterStore.
2. namespace: This identifies the Kubernetes namespace where our resources will be located.
3. team_id: It's used to denote the team that owns the particular resource. It is optional, but further it can be used for ABAC for AWS.
4. project_name: This is the name of the project under which the resources will fall.

These are used in our aws_ssm_parameter block to create secure parameters, which are initially set with placeholder values.

The remaining parameters, such as platform, fqdn, deployment_grace_period, svc_config, and hpa_config, although not explicitly used in our SSM parameters creation script, can be utilized within the same Terraform project to create various resources with AWS and Kubernetes providers. These could include load balancers, Horizontal Pod Autoscalers, and other vital components for our cloud infrastructure, and they contribute to the flexibility and comprehensiveness of the system we are setting up.

The aws_ssm_parameter block creates the SSM parameters. Each SSM parameter is given a placeholder value to initialize it. The actual values will be input later by a dedicated team member, such as a developer or a DevOps engineer, with sufficient permissions. This can be done via the AWS console or command-line interface (CLI).

It’s important to note that storing these values directly from the Terraform project is not advisable because they would end up in the Terraform state. This is a situation we want to avoid for security reasons, as we don’t want sensitive information like secrets stored in the Terraform state.

---

InitContainer with GO binary

Moving on to the next crucial step, we need to prepare the init container Golang binary. This binary will fetch our secrets from the AWS Parameter Store and write them into a file. Kubernetes will then load these secrets from the file into environment variables for use by our applications.

For this, we’ll write a small program in Go. The script’s operation can be summarized as follows: It creates an AWS session, uses the SSM client to fetch the secrets we stored earlier, writes the secrets into an environment file (.env), and then stores this file in a specific path (/etc/ssm/). The environment variables stored in this file will be available to other containers in the pod once loaded.

main.go

package main

import (
 "bufio"
 "fmt"
 "github.com/aws/aws-sdk-go/aws"
 "github.com/aws/aws-sdk-go/aws/session"
 "github.com/aws/aws-sdk-go/service/ssm"
 "os"
 "path"
)

func main() {
 sess, err := session.NewSession(&aws.Config{
  Region: aws.String(os.Getenv("AWS_REGION")),
 })

 if err != nil {
  fmt.Println("Failed to create session,", err)
  return
 }

 ssmSvc := ssm.New(sess)

 eks_cluster := os.Getenv("EKS_CLUSTER")
 namespace := os.Getenv("NAMESPACE")
 ci_project_name := os.Getenv("CI_PROJECT_NAME")

 ssmPath := fmt.Sprintf("/eks/%s/%s/%s", eks_cluster, namespace, ci_project_name)

 paramDetails := &ssm.GetParametersByPathInput{
  Path:           aws.String(ssmPath),
  WithDecryption: aws.Bool(true),
 }

 resp, err := ssmSvc.GetParametersByPath(paramDetails)
 if err != nil {
  fmt.Println("Failed to get parameters,", err)
  return
 }

 file, err := os.Create("/etc/ssm/.env")
 if err != nil {
  fmt.Println("Failed to create file,", err)
  return
 }
 defer file.Close()

 writer := bufio.NewWriter(file)
 for _, param := range resp.Parameters {
  name := path.Base(*param.Name)
  value := *param.Value
  writer.WriteString(fmt.Sprintf("export %s=%s\n", name, value))
 }

 err = writer.Flush()
 if err != nil {
  fmt.Println("Failed to write to file,", err)
 }
 fmt.Println("env file created successfully")
}

Dockerfile

FROM golang:1.20-alpine as BUILD
WORKDIR /app
COPY . .
RUN go build -o main

FROM alpine:3.16 AS RUNTIME
WORKDIR /app
COPY --from=BUILD /app/main .
CMD ["./main"]

In this Go script:

• We start by creating a new AWS session and initializing an SSM client using the aws-sdk-go package.
• We retrieve the environment variables for the EKS cluster, the namespace, and the project name. These will be used to construct the path to our secrets stored in the SSM Parameter Store.
• With the SSM client, we fetch the secrets from the Parameter Store using the GetParametersByPath method. This method retrieves all parameters within the provided path.
• We then create a .env file and write the fetched parameters into this file. Each line in the file will contain one secret, with the syntax export SECRET_NAME=secret_value.

Great, with the init container’s Go script ready, the next step is building this into a Docker image and pushing it to the Amazon Elastic Container Registry (ECR). Ideally, this should be part of your CI/CD process. Here’s a condensed guide to help you achieve this manually:

# Build the image, authenticate Docker to ECR, and push the image
docker build -t ssm-init-container . 
aws ecr get-login-password | docker login --username AWS --password-stdin your-account-id.dkr.ecr.region.amazonaws.com
aws ecr create-repository - repository-name ssm-init-container
docker tag ssm-init-container:latest your-account-id.dkr.ecr.region.amazonaws.comssm-init-container:latest 
docker push your-account-id.dkr.ecr.region.amazonaws.com/ssm-init-container:latest

Please replace ‘your-account-id’ and ‘region’ with your AWS account ID and your region, respectively.

---

OIDC Federated Access for EKS Pods

Before we proceed to actual deployments, we need to ensure the IAM roles are correctly associated with the Kubernetes service accounts. This is vital for the secure operation of our applications, allowing them to access necessary AWS resources.

Our Terraform scripts will take care of this association using the concept of IAM Roles for Service Accounts (IRSA) in AWS EKS, facilitated by OpenID Connect (OIDC) federation. It’s a recommended best practice to follow to ensure fine-grained access control to AWS services, directly from within the EKS environment. This eliminates the need to provide broad access permissions at the node level and greatly enhances our application’s security.

For enhanced security, AWS EKS allows IAM roles to be assigned to Kubernetes Service Accounts through a feature called IAM Roles for Service Accounts (IRSA). This mechanism uses OpenID Connect (OIDC) federation to drive the mapping between Kubernetes service accounts and AWS IAM roles. This section shows how to implement it using Terraform.

The following guides from AWS EKS are recommended reading to gain a deeper understanding of the topic:

1. Enabling IAM roles for service accounts on your cluster
2. Creating an IAM role and policy for your service account

Here’s the Terraform code that sets up the IAM roles, policies and associates them with Kubernetes service accounts:

# Assume Role Policy for service accounts
data "aws_iam_policy_document" "service_account_assume_role" {
  for_each = local.projects
  statement {
    actions = ["sts:AssumeRoleWithWebIdentity"]
    effect  = "Allow"
    condition {
      test     = "StringEquals"
      variable = "${replace(aws_iam_openid_connect_provider.oidc_provider_sts.url, "https://", "")}:aud"
      values   = ["sts.amazonaws.com"]
    }
    condition {
      test     = "StringEquals"
      variable = "${replace(aws_iam_openid_connect_provider.oidc_provider_sts.url, "https://", "")}:sub"
      values   = ["system:serviceaccount:${each.value.namespace}:${each.key}"]
    }
    principals {
      identifiers = [aws_iam_openid_connect_provider.oidc_provider_sts.arn]
      type        = "Federated"
    }
  }
}

# SSM Access permissions
resource "aws_iam_role" "service_account_role" {
  for_each           = local.projects
  assume_role_policy = data.aws_iam_policy_document.service_account_assume_role[each.key].json
  name               = "project-${each.key}-service-account-role"
  tags               = local.default_tags
}

# IAM Policy for SSM secrets
data "aws_iam_policy_document" "ssm_secret_policy" {
  for_each = local.projects
  statement {
    effect = "Allow"
    actions = ["ssm:GetParametersByPath", "ssm:GetParameters"]
    resources = [
      "arn:aws:ssm:${local.region}:${local.account_id}:parameter/eks/${var.cluster_name}/${each.value.namespace}/${each.key}*"
    ]
  }
}

resource "aws_iam_policy" "ssm_secret_policy" {
  for_each    = local.projects
  name        = "project-${each.key}-ssm-access"
  description = "Policy to allow EKS pods/projects to access respective SSM parameters"
  policy      = data.aws_iam_policy_document.ssm_secret_policy[each.key].json
  tags        = local.default_tags
}

resource "aws_iam_role_policy_attachment" "service_account_role_ssm" {
  for_each   = local.projects
  role       = aws_iam_role.service_account_role[each.key].name
  policy_arn = aws_iam_policy.ssm_secret_policy[each.key].arn
}

This code sets up an IAM role for each service account, granting it permissions to access specific SSM parameters. The service account is then mapped to the IAM role using OIDC. It’s a great approach to securely handle permissions and access secrets in EKS environments.

Remember to replace the placeholders with actual values before running the script. Make sure you also have the appropriate AWS access and permissions to execute these commands.

With this setup, your applications running in Kubernetes can securely and efficiently access the resources they need to function, all while adhering to the principle of least privilege.

---

Application Deployment

With our Go-based init container built and securely stored in ECR, let’s move on to deploying a demo application. For this, we’ll need an entrypoint.sh shell script, a Dockerfile for our application, and a Kubernetes Deployment manifest.

entrypoint.sh

FROM golang:1.20-alpine as BUILD
WORKDIR /build
COPY . .
RUN go mod tidy
RUN go build -o main
FROM alpine:3.16 AS RUNTIME
WORKDIR /app
COPY - from=BUILD /build .
RUN chmod +x entrypoint.sh
ENTRYPOINT ["./entrypoint.sh"]

To give you a head start, here is a simple Go application you can deploy for testing purposes: Go Resume Demo

Lastly, we have our Kubernetes Deployment manifest

apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: ${NAMESPACE}
  name: ${PROJECT_NAME}
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: ${PROJECT_NAME}
  replicas: 1
  template:
    metadata:
      labels:
        app.kubernetes.io/name: ${PROJECT_NAME}
        app.kubernetes.io/environment: ${EKS_CLUSTER}
        app.kubernetes.io/owner: Devops
    spec:
      serviceAccountName: ${PROJECT_NAME}
      initContainers:
        - name: secret-gopher
          image: ${ECR_REGISTRY}/<INIT_CONTAINER_NAME>:latest
          env:
            - name: EKS_CLUSTER
              value: ${EKS_CLUSTER}
            - name: NAMESPACE
              value: ${NAMESPACE}
            - name: CI_PROJECT_NAME
              value: ${PROJECT_NAME}
          volumeMounts:
            - name: envfile
              mountPath: /etc/ssm/
              subPath: .env
      containers:
        - image: ${BUILD_IMAGE}:${BUILD_TAG}
          volumeMounts:
            - name: envfile
              mountPath: /etc/ssm/
              subPath: .env
          imagePullPolicy: Always
          name: ${PROJECT_NAME}
          ports:
            - containerPort: 80
      volumes:
        - name: envfile
          emptyDir: {}

This sets us up for a Kubernetes deployment, which uses the init container we built previously to populate our environment variables from AWS SSM, securely managing our application’s secrets.

Here’s how it works:

1. When the Kubernetes Pod starts, the init container is the first to run. It fetches the secret data from AWS SSM and writes it to a file named .env located in the /etc/ssm/ directory.
2. This directory is a shared volume (emptyDir) that's accessible to all containers in the Pod. The emptyDir volume is created when a Pod is assigned to a Node, and it exists as long as that Pod is running on that Node. The data in emptyDir is safe across container crashes and restarts.
3. Once the init container successfully writes the .env file, it exits, and the main application container starts.
4. The main application container reads the .env file from the shared volume, thus having access to the secret data. The entrypoint.sh script in the main container sources the .env file to set the environment variables. Then, it removes the .env file for added security, ensuring the secrets do not persist in the file system once they've been loaded into the application's environment.
5. The main application then continues to run with the environment variables securely set, all without the secrets having to be explicitly stored or exposed outside the application’s memory.

The following block of code can be added to your application to debug and verify that you are retrieving the secret parameters correctly from the AWS Systems Manager (SSM) Parameter Store.

// Debug ssm fetching 
key1 := os.Getenv("AMQ_USER")
key2 := os.Getenv("AMQ_PASS")

filePath := "/app/ssm-vars"

file, err := os.Create(filePath)
if err != nil {
    log.Fatalf("Failed to create file: %v", err)
}
defer file.Close()

content := fmt.Sprintf("KEY1=%s\nKEY2=%s\n", key1, key2)
err = os.WriteFile(filePath, []byte(content), 0644)
if err != nil {
    log.Fatalf("Failed to write file: %v", err)
}

This piece of code reads the values of two environment variables, AMQ_USER and AMQ_PASS, which have been populated from the SSM Parameter Store. It then writes these values to a file for debugging purposes.

It’s important to understand that in a production environment, writing secrets to a file may expose them to additional security risks. This should only be used for debugging purposes and removed from the production code.

We achieved this secure management of secrets by following these steps:

1. Storing Secrets in AWS SSM Parameter Store: We stored our secrets securely in the AWS Systems Manager Parameter Store, which is a managed service that provides a secure and scalable way to store and manage configuration data.
2. Fetch Secrets with an Init Container: We used an init container in our Kubernetes pod, which runs before our main application starts. This init container runs a Go program that fetches the secrets from the AWS SSM Parameter Store and writes them to an environment file.
3. Populate Environment Variables: In the main container where our application runs, we used an entrypoint script that sources the environment file, thereby populating the environment variables with the secrets.
4. Removal of .env file: To ensure that our secrets are not written to disk in the main container, the entrypoint script removes the .env file after sourcing it.
5. Secrets in Memory: As a result of these steps, the secrets are only present in the memory of the running application process and nowhere else. They are not written to disk in the main container, and are not included in any container layers. This is a robust way to keep secrets secure.
6. Security Best Practices: We also ensured security best practices such as setting appropriate IAM roles for access to AWS SSM Parameter Store, and used Kubernetes RBAC for restricting access to Kubernetes

---

Farewell 😊

We’ve explored an effective method of Kubernetes secret management using Golang, AWS ParameterStore, OIDC, and Terraform. Thank you for reading this article. I hope it has provided you with valuable insight and can serve as a handy reference for your future projects. Keep exploring, keep learning, and continue refining your security practices!

Comments

[Ivan]

Hi! but I am following your post, I have a question, how do you access the ECR registry to download the image into the init container? pd: great post, thanks

[Tarlan Huseynov]

Hi Ivan, ty!
In eks environment, you depending on your specific needs, you would have either "EKS Node role" or if you're using fargate that would be "Fargate pod execution role".
These Service roles ( instance profile in case of Node role ) will need access permissions (iam policies) in place in order to be able to fetch private ecr images. This instance profile would seamlessly allow the kubelet fetch required images.
here is some reference:
docs.aws.amazon.com/eks/latest/use...
docs.aws.amazon.com/eks/latest/use...