DEV Community

Derrick Amenuve
Derrick Amenuve

Posted on

Storing Kamal secrets in AWS Secrets Manager and deploying to a cheap Hetzner VPS

#rails #kamal #devops #deployment

I ran into a problem with Kamal. My .kamal/secrets file was full of API keys sitting in plaintext on my laptop. Anyone with access could read them all.

TLDR; Use Kamal with AWS Secrets Manager and deploy to a Hetzner VPS. No plaintext secrets, cheap hosting, compliance happy.

The problem

Kamal is great for deploying apps. But by default secrets are in a plaintext file. For SOC 2 and GDPR that does not work. You need a managed store. I went with AWS Secrets Manager.

But then I hit another issue. The kamal secrets fetch --adapter aws_secrets_manager command with --from expects each key to be its own AWS secret. If you store everything as one JSON blob (like I did), you get:

ERROR (RuntimeError): myapp/production/secrets//DEEPGRAM_API_KEY: Secrets Manager can't find the specified secret.
Enter fullscreen mode Exit fullscreen mode

Step 1: Hetzner VPS

Hetzner CAX series starts at around 4 euro a month. I use the CX22 with 2 vCPUs and 4GB RAM. Enough for production.

# On your Hetzner server
apt update && apt install -y docker.io

# Copy your SSH key so Kamal can connect
ssh-copy-id root@your-server-ip
Enter fullscreen mode Exit fullscreen mode

Your config/deploy.yml:

servers:
  web:
    hosts:
      - runtime.yourdomain.com

proxy:
  ssl: true
  hosts:
    - runtime.yourdomain.com
  healthcheck:
    path: /health/ready

registry:
  server: docker.io
  username: your-docker-user
  password:
    - KAMAL_REGISTRY_PASSWORD
Enter fullscreen mode Exit fullscreen mode

You need a Docker Hub account and a personal access token for KAMAL_REGISTRY_PASSWORD.

Step 2: Create the secret in AWS

In the AWS Secrets Manager Console:

  1. Go to Secrets Manager > Store a new secret
  2. Select "Other type of secret"
  3. Switch to plaintext tab and paste your JSON 
{
  "DEEPGRAM_API_KEY": "your_deepgram_key",
  "ASSEMBLY_AI_API_KEY": "your_assemblyai_key",
  "REDIS_URL": "redis://:password@your-redis:6379",
  "KAMAL_REGISTRY_PASSWORD": "your_docker_token"
}
Enter fullscreen mode Exit fullscreen mode
  1. Name it myapp/production/secrets
  2. Click Store

Pick a region close to your server. If your Hetzner box is in Germany, use eu-central-1 (Frankfurt). Keeps latency low and GDPR happy.

Step 3: IAM user for your laptop

Your laptop needs permission to read the secret during deploy.

  1. Go to IAM > Users > Create user
  2. Name it kamal-deploy
  3. Uncheck console access (CLI only)
  4. Create a group called secrets-manager with the SecretsManagerReadWrite policy
  5. Add an inline policy for batch reading: 
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "secretsmanager:GetSecretValue",
        "secretsmanager:DescribeSecret",
        "secretsmanager:BatchGetSecretValue",
        "secretsmanager:ListSecrets"
      ],
      "Resource": "*"
    }
  ]
}
Enter fullscreen mode Exit fullscreen mode
  1. Add your user to the group

IAM policies can take a minute to propagate. If it fails at first, wait 30 seconds and try again.

Step 4: Configure AWS CLI 

aws configure
# AWS Access Key ID: paste from IAM user
# AWS Secret Access Key: paste
# Default region name: eu-central-1
# Default output format: json
Enter fullscreen mode Exit fullscreen mode

Test it:

aws secretsmanager get-secret-value --secret-id myapp/production/secrets --query SecretString --output text | head -c 50
Enter fullscreen mode Exit fullscreen mode

You should see the start of your JSON.

Step 5: Format your .kamal/secrets file

This is where I got stuck. The --from flag wants one AWS secret per key. Having 20 separate secrets is annoying. Check the Kamal secrets docs for more on this.

Instead I use the AWS CLI with Python extraction. Each line is self contained:

# AWS Secrets Manager: myapp/production/secrets (eu-central-1)
DEEPGRAM_API_KEY=$(python3 -c "import json,sys; print(json.loads(sys.argv[1])['DEEPGRAM_API_KEY'])" "$(aws secretsmanager get-secret-value --secret-id myapp/production/secrets --query SecretString --output text)")
ASSEMBLY_AI_API_KEY=$(python3 -c "import json,sys; print(json.loads(sys.argv[1])['ASSEMBLY_AI_API_KEY'])" "$(aws secretsmanager get-secret-value --secret-id myapp/production/secrets --query SecretString --output text)")
REDIS_URL=$(python3 -c "import json,sys; print(json.loads(sys.argv[1])['REDIS_URL'])" "$(aws secretsmanager get-secret-value --secret-id myapp/production/secrets --query SecretString --output text)")
KAMAL_REGISTRY_PASSWORD=$(python3 -c "import json,sys; print(json.loads(sys.argv[1])['KAMAL_REGISTRY_PASSWORD'])" "$(aws secretsmanager get-secret-value --secret-id myapp/production/secrets --query SecretString --output text)")
Enter fullscreen mode Exit fullscreen mode

Each line fetches the full JSON and extracts one key. Kamal evaluates each line in its own subshell so there are no shared variables between lines. This works.

You can also use jq if you prefer:

DEEPGRAM_API_KEY=$(aws secretsmanager get-secret-value --secret-id myapp/production/secrets --query SecretString --output text | jq -r '.DEEPGRAM_API_KEY')
Enter fullscreen mode Exit fullscreen mode

Step 6: Deploy 

kamal deploy
Enter fullscreen mode Exit fullscreen mode

Kamal fetches secrets from AWS during deploy and injects them into your container. No plaintext file ever touches the server.

Production and staging

I use a different AWS secret per environment. Both pull from AWS no plaintext anywhere.

# .kamal/secrets  (used by kamal deploy)
DEEPGRAM_API_KEY=$(python3 -c "import json,sys; print(json.loads(sys.argv[1])['DEEPGRAM_API_KEY'])" "$(aws secretsmanager get-secret-value --secret-id myapp/production/secrets --query SecretString --output text)")
KAMAL_REGISTRY_PASSWORD=$(python3 -c "import json,sys; print(json.loads(sys.argv[1])['KAMAL_REGISTRY_PASSWORD'])" "$(aws secretsmanager get-secret-value --secret-id myapp/production/secrets --query SecretString --output text)")

# .kamal/secrets.staging  (used by kamal deploy -d staging)
DEEPGRAM_API_KEY=$(python3 -c "import json,sys; print(json.loads(sys.argv[1])['DEEPGRAM_API_KEY'])" "$(aws secretsmanager get-secret-value --secret-id myapp/staging/secrets --query SecretString --output text)")
KAMAL_REGISTRY_PASSWORD=$(python3 -c "import json,sys; print(json.loads(sys.argv[1])['KAMAL_REGISTRY_PASSWORD'])" "$(aws secretsmanager get-secret-value --secret-id myapp/staging/secrets --query SecretString --output text)")
Enter fullscreen mode Exit fullscreen mode

Only the secret name changes between files. myapp/production/secrets for production, myapp/staging/secrets for staging. Run kamal deploy -d staging and Kamal reads from the staging file.

Both secrets live in AWS. No staging credentials in plaintext either. This matters for SOC 2 because auditors check every environment.

Done

No more secrets in plaintext. SOC 2 and GDPR requirements met. Hetzner bill stays under 5 euro a month.

Big thanks to the AWS docs team, the Kamal maintainers, and Hetzner for keeping hosting affordable. Hope this saves you the same headaches I ran into. Now back to building.

Top comments (0)