DEV Community: Rajesh Kumar

How to Set Up HashiCorp Vault on Kubernetes

Rajesh Kumar — Wed, 29 Jan 2025 04:04:20 +0000

Vault by HashiCorp is a powerful tool for managing secrets securely. In this tutorial, we’ll walk through setting up Vault on Kubernetes and using it with External Secrets to securely manage secrets in your applications.

Prerequisites:

Before we begin, ensure you have:

A running Kubernetes cluster.
kubectl installed and configured.
Helm installed on your system.

Step 1: Add the HashiCorp Helm Repository
HashiCorp provides a Helm chart for installing Vault. First, we need to add their repository to Helm:

$ helm repo add hashicorp https://helm.releases.hashicorp.com
$ helm repo update

This command adds HashiCorp’s Helm repository and updates your local Helm chart index.

Step 2: Install Vault Using Helm
Now, let’s deploy Vault into your Kubernetes cluster:

$ helm install vault hashicorp/vault -n vault --create-namespace

# vault: The name of the Helm release.
# hashicorp/vault: Specifies the Vault Helm chart from the HashiCorp repository.
# -n vault: Deploys Vault in the vault namespace.
# --create-namespace: Ensures the vault namespace is created if it doesn’t already exist.

Once this command runs successfully, Vault will be deployed, but it won’t be ready to use until initialized and unsealed.

Step 3: Initialize Vault
Vault must be initialized before it can store and manage secrets. Use the following command:

$ kubectl -n vault exec -it vault-0 -- vault operator init
This generates the unseal keys and an initial root token. Make sure to copy and securely store these tokens, as they are critical for accessing Vault.

You’ll see output like this:

Unseal Key 1: <key1>
Unseal Key 2: <key2>
Unseal Key 3: <key3>
Initial Root Token: <root-token>

Step 4: Unseal Vault
Vault operates in a “sealed” state by default. To make it operational, you need to “unseal” it using the keys generated earlier. Enter the Vault pod and unseal it:

$ kubectl -n vault exec -it vault-0 -- sh

Run the following commands to unseal Vault:

$ vault operator unseal
<insert key1>
$ vault operator unseal
<insert key2>
$ vault operator unseal
<insert key3>

You’ll paste each of the unseal keys when prompted. Once all keys are entered, the Vault becomes unsealed and operational.

Step 5: Log in to Vault
Log in using the initial root token:

$ vault login <INITIAL_ROOT_TOKEN>

Replace with the token you saved earlier.

Step 6: Enable a Secret Engine
Vault organizes secrets using engines. Here, we’ll enable the KV (Key-Value) engine:

$ vault secrets enable --version=2 --path=kv kv

# --version=2: Specifies that we’re using version 2 of the KV engine.
# --path=kv: Sets the mount path for the engine.

Step 7: Add Secrets to Vault
Let’s add some secrets to Vault. For example:

vault kv put -mount=kv dev-blog adminUser=test
vault kv put -mount=kv dev-blog adminPass='password'

Here:

# -mount=kv: Refers to the KV engine we enabled earlier.
# dev-blog: The path under which the secrets are stored.
# adminUser and adminPass: The key-value pairs representing your secrets.

You can repeat this process for all the secrets your application needs.

Step 8: Install External Secrets
External Secrets bridges the gap between Kubernetes and external secret management systems like HashiCorp Vault, AWS Secrets Manager, Azure Key Vault, and others. Its primary purpose is to sync secrets from these external systems into Kubernetes-native secrets so that your applications can access them seamlessly.

In this setup, we’re using Vault as the secret management solution to securely store sensitive data. However, Kubernetes applications require secrets to be in the form of Kubernetes-native secrets. External Secrets acts as the intermediary that syncs secrets from Vault into Kubernetes.

Deploying External Secrets at this stage ensures that we have the required infrastructure to make Vault secrets accessible to our Kubernetes workloads. Once External Secrets is installed and configured, it will continuously monitor Vault for updates and keep Kubernetes secrets in sync with the external secret store.

This ensures that:

Secrets are always up-to-date.
Developers and applications don’t need to directly interact with Vault.
The system remains secure, as Vault handles the secure storage and lifecycle of secrets, while Kubernetes secrets provide easy access for workloads.

$ helm install external-secrets external-secrets/external-secrets -n external-secrets --create-namespace

This installs External Secrets in the external-secrets namespace.

Step 9: Create a Vault Token Secret in Kubernetes
External Secrets needs a Vault token to authenticate. Create a Kubernetes secret with the token:

apiVersion: v1
data:
  token: <your-token-base64>
kind: Secret
metadata:
  name: vault-token
  namespace: external-secrets
type: Opaque

Replace with the Base64-encoded token (use echo -n "token" | base64 to generate it).

Save this file as vaultTokenSecret.yaml and apply it:

$ kubectl -n external-secrets apply -f vaultTokenSecret.yaml

Step 10: Configure External Secrets to Access Vault

Now, create a ClusterSecretStore resource to configure External Secrets to communicate with Vault:

apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: vault-backend-access
spec:
  provider:
    vault:
      server: "https://vault.rajesh-kumar.in/"
      path: "kv"
      version: "v2"
      auth:
        tokenSecretRef:
          name: "vault-token"
          key: "token"
          namespace: external-secrets

# server: The Vault server URL.
# path: The KV engine path (kv in our setup).
# auth.tokenSecretRef: Points to the secret containing the Vault token.

Save this as clusterSecretStore.yaml and apply it:

$ kubectl -n vault apply -f clusterSecretStore.yaml

Step 11: Verify and Use Secrets
With everything configured, you can create ExternalSecret resources to sync Vault secrets into Kubernetes secrets. For example:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: dev-blog-secrets
  namespace: dev-blog-ns
spec:
  secretStoreRef:
    name: vault-backend-access
    kind: ClusterSecretStore
  target:
    name: dev-blog-k8s-secret
  data:
  - secretKey: ADMIN_USER
    remoteRef:
      key: kv/dev-blog
      property: adminUser
  - secretKey: ADMIN_PASS
    remoteRef:
      key: kv/dev-blog
      property: adminPass

Apply this file, and External Secrets will create an externalsecrets resource named dev-blog-secrets and a Kubernetes secret named dev-blog-k8s-secret in the dev-blog-ns namespace with the values pulled from Vault.

$ kubectl get externalsecret -n medium-blog-ns -o yaml
$ kubectl get secret -n medium-blog-ns -o yaml

Bonus: Expose Vault with Ingress Using Traefik
To make Vault accessible externally, we’ll expose it using an Ingress. This setup ensures that Vault is accessible through a friendly domain name (vault.rajesh-kumar.in) using HTTP.

Step 1: Ensure Traefik is Deployed
Traefik is a popular Kubernetes Ingress Controller that provides robust routing capabilities. If you don’t have Traefik installed, follow these steps:

Add the Traefik Helm Repository:

$ helm repo add traefik https://traefik.github.io/charts helm repo update

Install Traefik: Deploy Traefik to your cluster:

$ helm install traefik traefik/traefik --namespace traefik --create-namespace

Verify Installation: Ensure that Traefik is running:

$ kubectl get pods -n traefik

You should see pods like traefik- running successfully.

Expose Traefik’s Dashboard (Optional): To access Traefik’s dashboard, configure an additional IngressRoute (optional but useful for debugging).

Step 2: Expose Vault Using Ingress
Create a IngressRoute resource to expose Vault using Traefik. Save the following configuration as:vaultIngressRoute.yaml

apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  name: vault-ingressroute
  namespace: vault
spec:
  entryPoints:
    - web
  routes:
  - match: Host(`vault.rajesh-kumar.in`)
    kind: Rule
    services:
    - name: vault
      port: 8200
      scheme: http

Explanation:
entryPoints: web: Indicates that this route uses the HTTP entry point. You can modify this to websecure for HTTPS if TLS is configured.
Host(vault.rajesh-kumar.in): The domain that maps to the Vault service. Ensure this domain points to your Kubernetes cluster via DNS.
services: name: vault: Refers to the Vault service created by the Helm chart.
port: 8200: The default port Vault listens on.

Step 3: Apply the IngressRoute
Apply the IngressRoute configuration to your cluster:

$ kubectl apply -f vaultIngressRoute.yaml

Step 4: Update DNS or Hosts File
Point your domain (vault.rajesh-kumar.in) to the external IP of the Traefik LoadBalancer. If you’re testing locally, you can edit your /etc/hosts file to map the domain to the NODE IP:

<NODE-IP> vault.rajesh-kumar.in

Step 5: Access Vault
Now you should be able to access Vault in your browser using http://vault.rajesh-kumar.in

Optional: Secure with HTTPS
For production environments, it’s highly recommended to secure the Ingress with TLS. To enable HTTPS:

Configure a certificate (e.g., via Let’s Encrypt or a self-signed certificate).
Update the Traefik entry point to websecure.
Modify the IngressRoute to include a tls section.
Example:

spec:
  entryPoints:
    - websecure
  tls:
    secretName: mysecret-cert
  routes:
  - match: Host(`vault.rajesh-kumar.in`)
    kind: Rule
    services:
    - name: vault
      port: 8200
      scheme: http

This ensures secure communication with Vault over HTTPS.

By exposing Vault with Ingress, you now have an external, accessible endpoint for managing secrets securely. This configuration is flexible, allowing further customization like TLS and additional routing rules to suit your needs.

Wrapping Up

Congratulations! You’ve successfully set up HashiCorp Vault on Kubernetes, stored secrets, and integrated it with External Secrets for secure secret management in Kubernetes.

This setup provides a robust foundation for securely managing application secrets, ensuring compliance, and enhancing security. Feel free to expand and customize this configuration for your specific use case!

Happy learning!

Thank you for reading!
Feel free to explore my articles for fascinating perspectives and useful suggestions. Let’s connect on LinkedIn. I’m eager to hear your thoughts and delve deeper into any discussions!

How to setup AWS IAM Roles Anywhere: A Step-by-Step Guide

Rajesh Kumar — Wed, 29 Jan 2025 03:24:13 +0000

Are you managing cloud infrastructure but need a secure way to access AWS resources from outside the AWS environment? AWS IAM Roles Anywhere has you covered! This feature enables you to assume IAM roles using X.509 certificates, eliminating the need for long-term AWS credentials. Let’s dive into how you can set this up, step by step.

What is AWS IAM Roles Anywhere?

Imagine you’re working with on-premises servers, CI/CD pipelines, or even IoT devices, and you need these to interact securely with AWS services. IAM Roles Anywhere allows these external systems to assume IAM roles without storing static access keys. It’s like giving your non-AWS workloads temporary AWS credentials but with extra layers of security.

Why Create Your Own Certificate Authority (CA)?

AWS provides services like Private Certificate Manager (ACM Private CA) to manage certificates, but these services can be expensive for smaller setups or learning environments. Creating your own CA gives you control over the certificate lifecycle without incurring additional AWS costs.

A Certificate Authority (CA) is the foundation of trust in a secure system. It issues certificates that prove the identity of your workloads. By creating your own Root CA, you establish a trusted entity that AWS can verify.

Prerequisites

Before we start, ensure you have the following:

AWS CLI installed and configured.
OpenSSL installed on your local machine.
Proper IAM permissions to create roles and trust anchors in AWS.
IAM role with the following permissions to manage Route 53:
IAM Policy for Route 53

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "route53:GetChange",
            "Resource": "arn:aws:route53:::change/*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "route53:ChangeResourceRecordSets",
                "route53:ListResourceRecordSets",
                "route53:ListHostedZones"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": "route53:ListHostedZonesByName",
            "Resource": "*"
        }
    ]
}

Trust Relationship for IAM Role

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "rolesanywhere.amazonaws.com"
            },
            "Action": [
                "sts:AssumeRole",
                "sts:SetSourceIdentity",
                "sts:TagSession"
            ]
        }
    ]
}

This IAM role is essential for enabling DNS management through Route 53 when creating a profile in AWS IAM Roles Anywhere.

Let’s Build It Step-by-Step:

Step 1: Create a Root Certificate Authority (CA)
The Root CA will be the anchor of trust for your setup. Here’s how to create it:

Generate the private key for your CA:

$ openssl genrsa -out privateRootCA.key 2048

# Explanation:

# genrsa: Generates an RSA private key.
# -out privateRootCA.key: Saves the private key to a file named privateRootCA.key.

Create a self-signed certificate for your Root CA:

$ openssl req -x509 -new -nodes -key privateRootCA.key -sha256 -days 365 -out privateRootCA.pem \
-subj "/C=IN/O=Learning DevOps" \
-addext "basicConstraints=critical,CA:TRUE" \
-addext "keyUsage=critical,keyCertSign,cRLSign"

# Explanation:

# req: Generates a certificate signing request or self-signed certificate.
# -x509: Creates a self-signed certificate instead of a CSR.
# -key privateRootCA.key: Uses the private key you just generated.
# -days 365: Makes the certificate valid for 365 days.
# -subj: Provides the certificate's subject details (country, organization, etc.).
# -addext: Adds certificate extensions, marking it as a CA capable of signing certificates.

This step creates two files:

privateRootCA.key: The private key for your Root CA (keep this secure).
privateRootCA.pem: The self-signed certificate that AWS will trust.

Step 2: Create a Trust Anchor in AWS
The trust anchor connects your Root CA with AWS IAM Roles Anywhere.

Upload the Root CA certificate to AWS:

$ aws rolesanywhere create-trust-anchor \
--name "MyTrustAnchor" \
--source "sourceType=CERTIFICATE_BUNDLE,sourceData={x509CertificateData=\"$(cat privateRootCA.pem)\"}"

# Explanation:

# create-trust-anchor: Creates a trust anchor in AWS.
# --source: Specifies the source type and data (your Root CA certificate).

Run this command to verify:

$ aws rolesanywhere list-trust-anchors

You should see your newly created trust anchor listed.

Step 3: Create a Client Certificate
Your external systems will use this certificate to authenticate.

Generate the private key for the client:

$ openssl genrsa -out client.key 2048

Generate the certificate signing request (CSR):

$ openssl req -new -key client.key -out client.csr -config <(\
  cat <<-EOF
  [req]
  default_bits = 2048
  prompt = no
  default_md = sha256
  distinguished_name = dn
  [dn]
  C = IN
  O = Learning DevOps
  CN = ClientCert
  [v3_ext]
  basicConstraints = CA:FALSE
  keyUsage = digitalSignature
  extendedKeyUsage = clientAuth
  EOF
)

# Explanation:

# -new: Creates a new CSR.
# -key client.key: Uses the client's private key.
# -config: Provides the certificate configuration inline.

Sign the client certificate with the Root CA:

$ openssl x509 -req -in client.csr -CA privateRootCA.pem -CAkey privateRootCA.key -CAcreateserial \
-out client.crt -days 365 -sha256 -extfile <(echo -e "basicConstraints = CA:FALSE\nkeyUsage = digitalSignature\nextendedKeyUsage = clientAuth")

# Explanation:

# x509: Creates an X.509 certificate.
# -CA privateRootCA.pem: Specifies the Root CA certificate.
# -CAkey privateRootCA.key: Specifies the Root CA private key.
# -CAcreateserial: Automatically generates a serial number for the client certificate.
# -out client.crt: Specifies the output client certificate file.

Step 4: Create a Profile in AWS
Profiles define which IAM roles external systems can assume.

$ aws rolesanywhere create-profile \
--name "TestProfile" \
--role-arns "arn:aws:iam::<AWS_ACCOUNT_ID>:role/<RoleName>" \
--trust-anchor-arn "arn:aws:rolesanywhere::<AWS_REGION>::trust-anchor/<TrustAnchorID>" \
--enabled

# Explanation:

# create-profile: Creates a profile in AWS.
# --role-arns: Arn of the role we created in the prerequisites section.
# --trust-anchor-arn: Links the profile to your trust anchor.

Verify your profiles:

$ aws rolesanywhere list-profiles

Step 5: Install AWS Signing Helper
The AWS Signing Helper simplifies signing API requests using your certificates.

Download and install it:

$ curl -O https://rolesanywhere.amazonaws.com/releases/1.4.0/X86_64/Linux/aws_signing_helper
$ chmod +x aws_signing_helper
$ sudo mv aws_signing_helper /usr/local/bin/

Step 6: Assume a Role Using the Client Certificate
Run the signing helper to fetch temporary credentials:

$ aws_signing_helper credential-process \
--trust-anchor-arn arn:aws:roles anywhere:ap-south-1:<Account_ID>:trust-anchor/<TrustAnchorID> \
--role-arn arn:aws:iam::<Account_ID>:role/<RoleName> \
--profile-arn arn:aws:rolesanywhere:ap-south-1:<Account_ID>:profile/<ProfileID> \
--certificate client.crt \
--private-key client.key

# Explanation:

# credential-process: Fetches temporary credentials for AWS.
# --certificate: Specifies the client certificate.
# --private-key: Specifies the client’s private key.

Export the credentials:

$ export AWS_ACCESS_KEY_ID=<AccessKeyIdFromOutput>
$ export AWS_SECRET_ACCESS_KEY=<SecretAccessKeyFromOutput>
$ export AWS_SESSION_TOKEN=<SessionTokenFromOutput>

Verify access:

$ aws route53 list-hosted-zones

Congratulations! You’ve securely accessed AWS resources from outside the AWS environment.

Wrapping Up

AWS IAM Roles Anywhere is a powerful tool for hybrid and on-premises setups. Securely linking your external workloads to AWS can eliminate static credentials and improve security.

Happy learning!