Securing Your Credentials with AWS Secrets Manager and KMS: A Complete Guide

truc3651 — Thu, 13 Mar 2025 10:42:16 +0000

In today's digital landscape, protecting sensitive information is paramount. Whether you're managing database passwords, API keys, or other credentials, proper encryption is essential. In this guide, we'll explore how AWS Secrets Manager and Key Management Service (KMS) work together to secure your sensitive data, starting with fundamental cryptographic concepts and building to practical implementation.

Understanding the Cryptographic Foundations

Before diving into AWS services, let's establish a solid understanding of the core encryption mechanisms that power modern security systems.

Symmetric Encryption: Simple but Powerful

Symmetric encryption uses a single key for both encryption and decryption. Think of it like a physical key to your house - the same key locks and unlocks the door.

Strengths:

Speed: Symmetric algorithms are computationally efficient, making them ideal for encrypting large amounts of data
Simplicity: The underlying mathematical operations are relatively straightforward

Weaknesses:

Key distribution problem: How do you securely share the key with others who need it?
No inherent authentication: The encryption itself doesn't verify who encrypted the data

Asymmetric Encryption: The Public-Private Pair

Asymmetric encryption uses two mathematically related keys: a public key that can be freely shared and a private key that must be kept secret. Data encrypted with the public key can only be decrypted with the corresponding private key.

Strengths:

Solves the key distribution problem: You can freely share your public key
Perfect for establishing secure connections: Enables secure communication between parties who have never previously shared secrets

Weaknesses:

Computationally expensive: Much slower than symmetric encryption
Key size requirements: Requires longer keys to achieve the same security level as symmetric encryption

Envelope Encryption: The Best of Both Worlds

Envelope encryption combines symmetric and asymmetric encryption to leverage the strengths of both while mitigating their weaknesses. Here's how it works:

Generate a data encryption key (DEK) using symmetric encryption
Encrypt the actual data using this DEK
Encrypt the DEK using an asymmetric key or another symmetric key (called the key encryption key or KEK)
Store the encrypted DEK alongside the encrypted data

This approach provides:

Performance efficiency: The bulk of the data is encrypted using fast symmetric encryption
Simplified key management: You only need to manage the KEK securely, not every individual DEK
Enhanced security: Even if a single DEK is compromised, other encrypted data remains secure
Key rotation capability: You can re-encrypt the DEK with a new KEK without having to re-encrypt all the data

Real-World Example: TLS/SSL and Envelope Encryption

The Transport Layer Security (TLS) protocol, which secures most internet traffic today, is a perfect example of envelope encryption in action.

When you connect to a secure website (HTTPS), here's what happens:

Certificate Exchange: The server presents its digital certificate containing its public key
Session Key Generation: Your browser generates a random symmetric session key
Key Exchange: Your browser encrypts this session key using the server's public key
Secure Communication: All further communication is encrypted using the symmetric session key

This process elegantly solves the primary challenges of both encryption types:

Asymmetric encryption is used briefly for the secure exchange of the session key
The computationally efficient symmetric encryption handles the bulk data transfer
No pre-shared secrets are required to establish a secure connection

Let's visualize this process:

Implementing Envelope Encryption with AWS KMS and Secrets Manager

AWS Key Management Service (KMS) and Secrets Manager use envelope encryption to protect your sensitive data. Let's see how to implement this with AWS CLI commands.

Step 1: Generate a Data Key using KMS

First, we'll generate a data key that will be used to encrypt our secret:

aws kms generate-data-key \
    --key-id alias/my-kms-key \
    --key-spec AES_256 \
    --output json > data-key.json

This command returns both:

A plaintext version of the data key (to use for encryption)
An encrypted version of the data key (encrypted by the KMS key)

Let's extract both parts:

cat data-key.json | jq -r .Plaintext > data-key-plaintext.base64
cat data-key.json | jq -r .CiphertextBlob > data-key-encrypted.base64

Step 2: Encrypt Your Secret Using the Data Key

Now we'll use the data key to encrypt our actual secret:

# Decode the base64 data key
cat data-key-plaintext.base64 | base64 --decode > data-key-plaintext.bin

# Encrypt our secret using OpenSSL with the data key
echo "MyDatabasePassword123!" | openssl enc -aes-256-cbc -salt \
    -in /dev/stdin \
    -out secret-encrypted.bin \
    -pass file:./data-key-plaintext.bin

# For security, immediately remove the plaintext data key
rm data-key-plaintext.base64 data-key-plaintext.bin

Step 3: Store Both in AWS Secrets Manager

Now we need to store both the encrypted secret and the encrypted data key:

# Convert the encrypted secret to base64
base64 secret-encrypted.bin > secret-encrypted.base64

# Create a JSON structure for our secret
cat > secret-payload.json << EOF
{
  "encryptedSecret": "$(cat secret-encrypted.base64)",
  "encryptedDataKey": "$(cat data-key-encrypted.base64)"
}
EOF

# Store in Secrets Manager
aws secretsmanager create-secret \
    --name "my-database-credentials" \
    --secret-string "$(cat secret-payload.json)"

Step 4: Retrieving and Decrypting the Secret

When you need to retrieve and decrypt your secret:

# Get the secret from Secrets Manager
aws secretsmanager get-secret-value \
    --secret-id "my-database-credentials" > retrieved-secret.json

# Extract the components
cat retrieved-secret.json | jq -r '.SecretString' | jq -r '.encryptedDataKey' > retrieved-data-key.base64
cat retrieved-secret.json | jq -r '.SecretString' | jq -r '.encryptedSecret' > retrieved-secret.base64

# Decrypt the data key using KMS
aws kms decrypt \
    --ciphertext-blob fileb://<(cat retrieved-data-key.base64 | base64 --decode) \
    --output text \
    --query Plaintext > decrypted-data-key.base64

# Use the decrypted data key to decrypt the secret
cat retrieved-secret.base64 | base64 --decode > retrieved-secret.bin
cat decrypted-data-key.base64 | base64 --decode > decrypted-data-key.bin

openssl enc -aes-256-cbc -d -salt \
    -in retrieved-secret.bin \
    -pass file:./decrypted-data-key.bin

This will output your original secret: MyDatabasePassword123!

Simplifying with AWS Secrets Manager's Native Functionality

The steps above explicitly demonstrate the envelope encryption process. However, AWS Secrets Manager handles most of this complexity for you. Here's how to use it more directly:

# Store a secret more simply (KMS envelope encryption happens automatically)
aws secretsmanager create-secret \
    --name "my-simple-credential" \
    --secret-string "MyDatabasePassword123!" \
    --kms-key-id alias/my-kms-key

# Retrieve and automatically decrypt the secret
aws secretsmanager get-secret-value \
    --secret-id "my-simple-credential" \
    --query SecretString \
    --output text

Why Don't We Need to Provide the Data Key When Decrypting?

You may have noticed that when retrieving secrets, we don't need to manually provide the data key. This is because of how AWS implements envelope encryption:

The encrypted data key is stored alongside your encrypted data
When you request the secret, AWS automatically:
- Retrieves the encrypted data key
- Decrypts it using the KMS key (if you have permission)
- Uses the decrypted data key to decrypt your actual secret
- Returns the plaintext secret to you

AWS maintains a complete history of all the keys that have been used, allowing you to decrypt data even after key rotation. This key history management is handled by KMS, relieving you of this complex burden.

Conclusion

By using envelope encryption, AWS combines the performance benefits of symmetric encryption with the key management advantages of asymmetric approaches. This approach allows you to focus on building your applications while ensuring your secrets remain secure.

Blue/green deployment with helm hands-on

truc3651 — Thu, 19 Dec 2024 17:36:13 +0000

QuickCap

Kubernetes uses RollingUpdate as the default deployment strategy, which transition to new version gradually.

Blue/green deployment strategy is about there're always 2 replica versions of application, this makes it easier to rollback to the previous version, if something goes wrong with the new release, and of course it's ideal for stateless application only.

Foreword

This post is about give you an overview picture of how blue/green deployment. There's no good practices, and re-use code well. If it's your thing, coninue reading :))

Resource

Clone repo: https://github.com/truc3651/devops

Quick look at the structure

Folder app contains everything for helm to release our app.
app/templates: where application template resident (deployment, config, secret, service)
app/Chart.yaml: release information (app name, version)
app/values.yaml: default environment variables that helm will use when install/upgrade app

deployment.sh: contains job visualize how blue/green deployment

Get your hands dirty

Install helm, kind
We use kind as local Kubernetes clusters.

brew install helm kind

Create a default cluster

kind create cluster --name helm

We have 1 control-plane

Change mod script

chmod +x ./deployment.sh

First time deploy your application

./deployment.sh init

We just create a deployment with 1 pod (default version blue)

At the beginning, service points to pod has label app=app

Release a new version
After weeks, your development team want to release new features.

./deployment.sh deploy-green

You can see blue pod has been terminated, and a new green pod has been created, also green deployment created as well.

Now our service points to pod which has label color=green

Release a new version 3rd time
After many weeks, your team want to introduce bunch cool features

./deployment.sh deploy-new-blue

This time, green pod has been terminated, and a new blue pod has been created.

And our service points to pod which has label color=blue

Rollback
The very new version has some critical bugs, so our team decides to rollback to the previous version

./deployment.sh rollback

You can see now green alive

DEV Community: truc3651