Bitwarden Secrets Manager on EKS – Per-App Integration with Atlantis

Bitwarden Secrets Manager on EKS – Per-App Integration with Atlantis

Sync secrets from Bitwarden Secrets Manager into Kubernetes on EKS using the sm-operator, AWS Secrets Manager for the machine token, and the Secrets Store CSI Driver. This guide expands on the base integration with a per-app, per-namespace pattern and uses Atlantis as a concrete example. It covers Terraform, Kustomize overlays, Argo CD, sync waves, and troubleshooting.

Note: Use placeholder values for org IDs and secret IDs. Never commit real tokens. For production, follow least-privilege IAM and rotation practices.

---

1. Overview

What this guide does:

• Integrates Bitwarden Secrets Manager with EKS via sm-operator, AWS Secrets Manager, and Secrets Store CSI Driver
• Uses a per-app namespace pattern: each app (e.g. Atlantis) gets its own SecretProviderClass, bw-auth-token-sync, and BitwardenSecret in its own namespace
• Walks through Terraform (EKS + Pod Identity per app), manifests, Argo CD Applications, validation, and force-sync
• Uses Atlantis (Terraform PR automation) as a worked example with GitHub App credentials

Why per-app namespaces?

• Isolation: each app has its own Bitwarden machine token (scoped in AWS)
• Simplicity: the sm-operator creates the K8s Secret in the app’s namespace; no cross-namespace wiring
• Consistency: same pattern for every new app

---

2. Prerequisites

• EKS cluster with Secrets Store CSI Driver installed
• Argo CD installed
• Terraform-managed EKS (e.g. terraform-aws-eks-basic with Secrets Manager support)
• Bitwarden Secrets Manager organization with Machine Account
• Per-app AWS secret path: bitwarden/sm-operator/<app>/machine-token

---

3. Architecture Overview

Bitwarden SM (Machine Account + App Secrets)
        │
        │ machine token (per app: bitwarden/sm-operator/<app>/machine-token)
        ▼
AWS Secrets Manager
        │
        │ Pod Identity + CSI (in app namespace, e.g. atlantis-1)
        ▼
SecretProviderClass + bw-auth-token-sync → creates bw-auth-token
        │
        ▼
BitwardenSecret (authToken: bw-auth-token)
        │
        │ sm-operator fetches via Bitwarden API
        ▼
Output K8s Secret (e.g. atlantis-1-vcs) in app namespace

Flow (per app, e.g. atlantis-1):

1. Machine token stored in AWS Secrets Manager as bitwarden/sm-operator/<app>/machine-token (JSON: {"token":"<value>"}).
2. SecretProviderClass + bw-auth-token-sync Deployment in the app namespace: CSI Driver mounts the token and creates K8s Secret bw-auth-token.
3. BitwardenSecret in the same namespace: authToken.secretName: bw-auth-token. The sm-operator reads this token and calls the Bitwarden API.
4. sm-operator creates the output K8s Secret (e.g. atlantis-1-vcs) in the app’s namespace. Atlantis (or any app) uses it directly.

---

4. Bitwarden Setup

1. Machine Account and token: Bitwarden Admin → Machine Accounts → Create Access Token. Copy the token.
2. Per-app machine token (optional): For isolation, create a separate token per app and store in bitwarden/sm-operator/<app>/machine-token in AWS.
3. App secrets: In Bitwarden Secrets Manager, create the secrets your apps need. For Atlantis (GitHub App): webhook secret, private key (key.pem).
4. Copy IDs: From Bitwarden Admin → Settings → Organization, copy organizationId. For each secret, copy its bwSecretId (UUID).

---

5. Terraform

Requirement: EKS module must enable Secrets Manager with Pod Identity. Add an association per app namespace (e.g. { namespace = "atlantis-1", service_account = "awssm-sync" }) and prefix bitwarden/sm-operator.

Create the AWS Secrets Manager secret per app. Pass the token via -var or TF_VAR_. Never commit it.

variable "bitwarden_sm_machine_token_atlantis" {
  description = "Bitwarden SM machine token for atlantis-1"
  type        = string
  default     = "REPLACE_WITH_REAL_TOKEN"
  sensitive   = true
}

resource "aws_secretsmanager_secret" "bitwarden_atlantis" {
  name        = "bitwarden/sm-operator/atlantis-1/machine-token"
  description = "Bitwarden SM machine token for atlantis-1"
  tags        = var.tags
}

resource "aws_secretsmanager_secret_version" "bitwarden_atlantis" {
  secret_id     = aws_secretsmanager_secret.bitwarden_atlantis.id
  secret_string = jsonencode({ token = var.bitwarden_sm_machine_token_atlantis })
}

---

6. Per-App Manifests (Atlantis Example)

Each app gets its own overlay with:

• SecretProviderClass
• ServiceAccount + bw-auth-token-sync Deployment
• BitwardenSecret
• The app itself (e.g. Atlantis Helm chart)

6.1 SecretProviderClass

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: bitwarden-sm-token
  namespace: atlantis-1
  annotations:
    argocd.argoproj.io/sync-wave: "-1"
spec:
  provider: aws
  parameters:
    region: ap-southeast-2
    usePodIdentity: "true"
    objects: |
      - objectName: "bitwarden/sm-operator/atlantis-1/machine-token"
        objectType: "secretsmanager"
        jmesPath:
          - path: token
            objectAlias: token
  secretObjects:
    - secretName: bw-auth-token
      type: Opaque
      data:
        - objectName: token
          key: token

6.2 ServiceAccount + bw-auth-token-sync

apiVersion: v1
kind: ServiceAccount
metadata:
  name: awssm-sync
  namespace: atlantis-1
  annotations:
    argocd.argoproj.io/sync-wave: "-2"
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: bw-auth-token-sync
  namespace: atlantis-1
  annotations:
    argocd.argoproj.io/sync-wave: "-1"
  labels:
    app: bw-auth-token-sync
spec:
  replicas: 1
  selector:
    matchLabels:
      app: bw-auth-token-sync
  template:
    metadata:
      labels:
        app: bw-auth-token-sync
    spec:
      serviceAccountName: awssm-sync
      containers:
        - name: pause
          image: registry.k8s.io/pause:3.9
          resources:
            requests:
              cpu: 1m
              memory: 4Mi
          volumeMounts:
            - name: secrets-store
              mountPath: "/mnt/secrets-store"
              readOnly: true
      volumes:
        - name: secrets-store
          csi:
            driver: secrets-store.csi.k8s.io
            readOnly: true
            volumeAttributes:
              secretProviderClass: bitwarden-sm-token

6.3 BitwardenSecret

apiVersion: k8s.bitwarden.com/v1
kind: BitwardenSecret
metadata:
  name: atlantis-1-vcs
  namespace: atlantis-1
  annotations:
    argocd.argoproj.io/sync-wave: "0"
spec:
  organizationId: "REPLACE_ORG_ID"
  secretName: atlantis-1-vcs
  onlyMappedSecrets: true
  map:
    - secretKeyName: github_secret
      bwSecretId: "REPLACE_WEBHOOK_SECRET_ID"
    - secretKeyName: key.pem
      bwSecretId: "REPLACE_PRIVATE_KEY_ID"
  authToken:
    secretName: bw-auth-token
    secretKey: token

6.4 Atlantis Helm values

# application-patch.yaml
vcsSecretName: atlantis-1-vcs
githubApp:
  id: "YOUR_GH_APP_ID"
  installationId: "YOUR_GH_INSTALLATION_ID"
ingress:
  enabled: false   # Enable when you have ingress + atlantisUrl

Use argocd.argoproj.io/sync-wave: "5" on the Atlantis Application so it is created after the BitwardenSecret and bw-auth-token-sync.

---

7. Sync Waves (Ordering)

To avoid the app starting before secrets exist:

• -2: ServiceAccount awssm-sync
• -1: SecretProviderClass, bw-auth-token-sync Deployment
• 0: BitwardenSecret, ConfigMaps
• 5: Atlantis Application (Helm)

---

8. Validation

# Verify secrets in app namespace
kubectl get secret atlantis-1-vcs bw-auth-token -n atlantis-1

# BitwardenSecret status
kubectl get bitwardensecret atlantis-1-vcs -n atlantis-1
kubectl get bitwardensecret atlantis-1-vcs -n atlantis-1 \
  -o jsonpath='{.status.conditions[?(@.type=="SuccessfulSync")].status}'

# Output secret keys (should show github_secret, key.pem)
kubectl get secret atlantis-1-vcs -n atlantis-1 -o jsonpath='{.data}' | jq -r 'keys[]'

# Atlantis pod
kubectl get pods -n atlantis-1
kubectl logs -n atlantis-1 -l app=atlantis --tail=20

---

9. Local Access (Port-Forward)

The Atlantis Service exposes port 80 (targetPort 4141):

kubectl port-forward svc/atlantis-1 -n atlantis-1 4141:80

Then open http://localhost:4141.

---

10. Force Sync (Token Refresh)

When the machine token in AWS Secrets Manager changes:

# Per-app namespace
kubectl delete secret bw-auth-token -n atlantis-1
kubectl delete pod -n atlantis-1 -l app=bw-auth-token-sync --force --grace-period=0
kubectl wait --for=condition=ready pod -l app=bw-auth-token-sync -n atlantis-1 --timeout=60s
kubectl rollout restart deployment/sm-operator-controller-manager -n sm-operator-system
kubectl rollout restart statefulset atlantis-1 -n atlantis-1

---

11. Adding Another App

For each new app:

1. Create app directory: apps/<app>/overlays/<cluster>/
2. Add SecretProviderClass (AWS path: bitwarden/sm-operator/<app>/machine-token)
3. Add bw-auth-token-sync (SA + Deployment)
4. Add BitwardenSecret with the app’s secret mapping
5. Add Terraform: { namespace = "<app>", service_account = "awssm-sync" } + AWS secret
6. Deploy the app (Helm, etc.) with the created secret

---

12. Summary: Copy-Paste

Validation:

kubectl get secret atlantis-1-vcs bw-auth-token -n atlantis-1
kubectl get bitwardensecret atlantis-1-vcs -n atlantis-1 -o jsonpath='{.status.conditions[?(@.type=="SuccessfulSync")].status}'
kubectl get secret atlantis-1-vcs -n atlantis-1 -o jsonpath='{.data}' | jq -r 'keys[]'

Port-forward:

kubectl port-forward svc/atlantis-1 -n atlantis-1 4141:80

Force sync (token changed):

kubectl delete secret bw-auth-token -n atlantis-1
kubectl delete pod -n atlantis-1 -l app=bw-auth-token-sync --force --grace-period=0
kubectl wait --for=condition=ready pod -l app=bw-auth-token-sync -n atlantis-1 --timeout=60s
kubectl rollout restart deployment/sm-operator-controller-manager -n sm-operator-system
kubectl rollout restart statefulset atlantis-1 -n atlantis-1

---

13. Troubleshooting

Issue: CreateContainerConfigError – Secret atlantis-1-vcs missing

Solution: Wait for sm-operator to sync BitwardenSecret; or restart pod after secret exists.

Issue: Pod Identity / token association error

Solution: Add { namespace = "atlantis-1", service_account = "awssm-sync" } to secrets_manager_associations in Terraform.

Issue: ResourceNotFoundException – AWS secret missing

Solution: Create bitwarden/sm-operator/atlantis-1/machine-token in Secrets Manager (JSON: {"token":"<value>"}).

Issue: Error: --gh-app-id/--gh-app-key-file... – VCS secret not reaching container

Solution: Ensure vcsSecretName and githubApp.id/installationId in Helm values; verify atlantis-1-vcs exists.

Issue: Argo CD app stuck "Progressing"

Solution: Ingress without controller. Set ingress.enabled: false until ingress is configured.

---

14. References

• Bitwarden SM Operator
• Secrets Store CSI Driver
• Atlantis Helm Chart