⚡ TL;DR: Encrypting data is easy; the hard part is managing the keys. Use envelope encryption with a KMS or HSM so you control where keys live, who can use them, how they rotate, and who audits them.
Contents
- The hard part isn't encryption
- One sentence, then a picture
- Envelope encryption, drawn out
- Where keys live (and why it matters)
- Envelope-encrypt a payload
- At rest vs in transit, rotation, and audit
- Common mistakes that cost hours
- Takeaways
- Where to go next
The hard part isn't encryption
Encrypting a blob of data is a solved problem. Pick AES-256, call one library function, and you have ciphertext. The first time most teams meet encryption, it feels almost too easy. Then the real question arrives: where does the key live? Because ciphertext plus the key sitting right next to it is just a slightly slower way to store plaintext.
Every hard problem in cryptography in production is a key management problem. Who can use the key? How do you rotate it without re-encrypting petabytes? How do you prove, six months later during an audit, exactly who decrypted that customer record and when? This article is about the lifecycle of the key, not the math of the cipher. The cipher you can trust; the key you have to govern.
📌 Who this is for: Engineers who can already call an encryption library but have never run a Key Management Service (KMS) in anger. If you have ever hardcoded a key in a config file, set a key as an environment variable, or wondered what "envelope encryption" actually means, this is for you. We assume you have read HTTPS, TLS & Encryption Basics, that covers data in transit; this covers the keys themselves.
One sentence, then a picture
A KMS doesn't store your secrets, it stores the keys that protect your secrets, and never lets those keys leave the building.
The mental model for every cloud KMS
That last clause is the whole point. A well-designed KMS performs cryptographic operations on your behalf so the master key material never appears in your application's memory, your logs, or a developer's laptop. You send it data to wrap; it sends back wrapped data. The key stays inside the vault.
| In the real world | In tech |
|---|---|
| A master key locked in the hotel's vault, never carried around | The Customer Master Key (CMK), lives only inside KMS/HSM, never exported |
| The front desk signs out a single-use room key when you check in | KMS generates a fresh data key for each payload or session |
| The room key is destroyed at checkout; the master is untouched | The plaintext data key is discarded after use; only its wrapped form is kept |
| A log of which staff opened the vault and when | Every Encrypt/Decrypt call is recorded in the audit trail (CloudTrail, etc.) |
A KMS is a hotel's key desk, not a key on a hook.
Envelope encryption, drawn out
You almost never encrypt large data directly with the master key. The master key is slow to reach (a network call), rate-limited, and you want to minimize how often it touches anything. Instead you use envelope encryption: the master key encrypts a small data key, and the fast, local data key encrypts the actual payload. The diagram below shows both the encrypt and the decrypt path.
Envelope encryption: the master key wraps a data key; the data key encrypts the payload. Decrypt reverses it.
- Ask KMS for a data key: Call GenerateDataKey against your master key. KMS returns the data key twice: once in plaintext (for immediate use) and once wrapped (encrypted by the master key).
- Encrypt locally with the plaintext data key: Use the plaintext data key with a fast local AES cipher to encrypt your payload. This happens in your process, no per-byte network round-trip to KMS.
- Throw away the plaintext data key: Wipe it from memory the moment you are done. The only copy that survives is the wrapped one.
- Store ciphertext + wrapped key together: Persist the encrypted payload alongside the wrapped data key. You never store the plaintext key, so leaking the storage layer alone reveals nothing.
- To decrypt, unwrap first: Send the wrapped data key to KMS Decrypt. KMS uses the master key to return the plaintext data key, which you then use locally to decrypt the payload, then discard again.
Where keys live (and why it matters)
Not all key storage is equal. The single biggest jump in security comes from moving the key out of anything a developer or an attacker can read directly. Here is the spectrum, worst to best.
| Where | Security | Rotation | Notes |
|---|---|---|---|
| In code / config | Terrible, lives in Git history forever | Manual + redeploy | One leaked repo = full compromise. Never do this. |
| Env variable | Weak, readable via process dump, crash logs, CI | Manual restart | Better than code, still plaintext at rest in the platform. |
| KMS (managed) | Strong, key never leaves the service | Automatic, scheduled | Operations are audited and access-controlled. The default for most teams. |
| HSM (dedicated) | Strongest, tamper-resistant hardware, FIPS 140-2/3 | Automatic, policy-driven | For regulated workloads (payments, gov). Higher cost and ops overhead. |
Each step up shrinks who can see the key and makes rotation automatable.
A Hardware Security Module (HSM) is a physical (or cloud-hosted dedicated) device built so that key material physically cannot be extracted, attempts to tamper with it zero out the keys. A managed KMS is usually backed by HSMs you share with other tenants; a dedicated HSM (CloudHSM, Azure Dedicated HSM) gives you single-tenant hardware and full control of the key store, at real operational cost. Most teams should reach for managed KMS first and only graduate to a dedicated HSM when a compliance regime demands it.
Envelope-encrypt a payload
Here is the full envelope flow in Python against AWS KMS. The same shape applies to Azure Key Vault and Google Cloud KMS, only the SDK names change. Notice we never see the master key, and we wipe the plaintext data key after use.
envelope.py
import os
import boto3
from cryptography.hazmat.primitives.ciphers.aead import AESGCM
kms = boto3.client("kms")
MASTER_KEY_ID = os.environ["KMS_KEY_ID"] # an alias/ARN, NOT key material
def encrypt(plaintext: bytes) -> dict:
# 1. Ask KMS for a fresh data key (256-bit) wrapped by the master key.
resp = kms.generate_data_key(KeyId=MASTER_KEY_ID, KeySpec="AES_256")
plaintext_key = resp["Plaintext"] # use locally, then destroy
wrapped_key = resp["CiphertextBlob"] # safe to store alongside data
# 2. Encrypt locally with the plaintext data key (fast, no network).
nonce = os.urandom(12)
ciphertext = AESGCM(plaintext_key).encrypt(nonce, plaintext, None)
# 3. Drop the plaintext data key from memory.
del plaintext_key
return {"wrapped_key": wrapped_key, "nonce": nonce, "ciphertext": ciphertext}
def decrypt(envelope: dict) -> bytes:
# 4. Unwrap the data key, KMS uses the master key, then we discard it.
plaintext_key = kms.decrypt(CiphertextBlob=envelope["wrapped_key"])["Plaintext"]
plaintext = AESGCM(plaintext_key).decrypt(
envelope["nonce"], envelope["ciphertext"], None
)
del plaintext_key
return plaintext
💡 Tip: The data key is single-purpose and cheap to mint. Generate a new one per object (or per small batch) rather than reusing one across millions of records, it limits the blast radius if a single plaintext data key ever leaks.
At rest vs in transit, rotation, and audit
At rest vs in transit
Encryption at rest protects data sitting on disk, your database files, object storage, snapshots, backups. This is what KMS and envelope encryption are for: if someone walks off with the drive (or dumps the S3 bucket), they get ciphertext. Encryption in transit protects data moving across a network using TLS, so an eavesdropper on the wire sees nothing useful. You need both, they defend against different attackers. At-rest defends the stolen disk; in-transit defends the tapped wire. Covering one and skipping the other leaves a wide-open door. (For the in-transit half, see HTTPS, TLS & Encryption Basics.)
Key rotation
Keys should not live forever. Rotation means periodically retiring a key and creating a new one, so a key that leaks has a limited useful lifetime and you bound how much data any single key ever protected. The beauty of envelope encryption is that rotating the master key is cheap: old wrapped data keys can still be unwrapped by KMS (it keeps prior versions), so you do not have to re-encrypt your data to rotate the master. Managed KMS can rotate the master automatically on a schedule (e.g. yearly). Data keys, being per-object, rotate naturally as you write new data.
Access control and audit
A key is only as safe as the policy on it. Use IAM-style key policies to grant least privilege: this service may Encrypt, that service may Decrypt, and almost nobody may administer the key or schedule its deletion. Crucially, every operation against the key is logged. In AWS that means kms:Decrypt calls land in CloudTrail; Azure and GCP have equivalents. That audit trail is how you answer "who decrypted this record, from where, and when", and it is what an auditor will ask for. A key with no access policy and no audit log is a liability dressed up as a control.
Common mistakes that cost hours
- Keys in code or config. Once a key hits Git, it is in the history forever, rotating it means rewriting history and re-keying. Treat any key that touches a repo as already compromised.
- No rotation. A key that has protected three years of data is a single point of catastrophic failure. If it leaks, every byte it ever wrapped is exposed. Schedule rotation and let envelope encryption keep it cheap.
- One key for everything. A single master key shared across services, tenants, and environments means one compromised caller decrypts the whole company. Scope keys per environment and per data domain to contain the blast radius.
- No audit on key use. If you cannot say who decrypted what and when, you cannot detect abuse and you cannot pass an audit. Enable KMS call logging from day one, it is the cheapest control you will ever turn on.
Takeaways
The whole article in six lines
- Encryption is easy; key management is the hard, governable part.
- A KMS keeps the master key inside the vault and operates on your behalf, the key never leaves.
- Envelope encryption: the master key wraps a fast local data key; you store ciphertext + wrapped key, never the plaintext key.
- Move keys out of code/env and into KMS (or an HSM for regulated workloads).
- You need at rest and in transit, they stop different attackers.
- Rotate keys, scope them per domain, and audit every Encrypt/Decrypt call.
Where to go next
Key management is one pillar of cloud security. Pair it with the credential side and the in-transit side to cover the whole story.
- Secrets Management, the companion piece: managing application credentials (API keys, DB passwords) rather than the cryptographic keys themselves.
- HTTPS, TLS & Encryption Basics, how the in transit half works, end to end.
- Cloud Engineer path, where key management, IAM, and network security fit into the full role.
Originally published on TheSimplifiedTech, where this guide is interactive, with in-browser terminal labs and diagrams. Learn cloud and DevOps by doing, no videos.
Top comments (0)