The Problem
You've built a service that needs a secret at runtime. Maybe it's a keystore password, a signing key, or an API master secret. Today, one person knows that password. They type it in, the service unlocks, and everything works.
Then your team grows.
Now three people need to restart the service at 3 AM when it crashes. But you can't just share the master password — that's a security nightmare. If one person leaves the team, you'd have to rotate the master secret, re-encrypt everything, and coordinate the change across all admins.
You need a system where:
- Multiple admins can unlock the service using their own passwords
- No one ever sees or shares the actual master secret
- Revoking an admin doesn't require re-encrypting the master secret
- Adding a new admin doesn't require the master secret to be present in plaintext
Sound impossible? It's not. This is a solved problem called envelope encryption.
Why Naive Approaches Fail
Before we get to the solution, let's rule out the obvious ideas:
"Just share the password"
- No audit trail of who unlocked the service
- Can't revoke access without changing the password for everyone
- Violates the principle of least privilege
- One person writes it on a sticky note and you're done
"Store the password in a vault (e.g., HashiCorp Vault)"
- Now you need a password to unlock the vault
- Turtles all the way down — you've just moved the problem
- Adds infrastructure complexity and a new single point of failure
"Encrypt the master password separately for each admin"
- If the master password changes, you re-encrypt N copies
- Key management becomes O(N) for every rotation
- No clean separation between "who can authenticate" and "who can decrypt"
The Solution: Envelope Encryption
Envelope encryption uses a two-tier key hierarchy to cleanly separate authentication from secret access.
Here's the architecture:
╔═══════════════════════════════════════╗
║ MASTER SECRET ║
║ (e.g., keystore password) ║
╚═══════════════════╤═══════════════════╝
│
encrypted by DATA KEY
│
╔═══════════════════╧═══════════════════╗
║ DATA KEY ║
║ (random 32-byte AES key) ║
╚═══╤═══════════════╤═══════════════╤═══╝
│ │ │
│ Argon2id + │ Argon2id + │ AES-256-GCM
│ AES-256-GCM │ AES-256-GCM │ (raw key)
│ │ │
┌───────▼───────┐ ┌────▼────────┐ ┌────▼────────┐
│ Admin A's │ │ Admin B's │ │ Recovery │
│ encrypted │ │ encrypted │ │ encrypted │
│ data_key │ │ data_key │ │ master │
│ │ │ │ │ │
│ locked with │ │ locked with│ │ locked with│
│ password A │ │ password B │ │ offline key│
├───────────────┤ ├─────────────┤ ├─────────────┤
│ users row │ │ users row │ │ envelope_ │
│ (DB) │ │ (DB) │ │ master (DB) │
└───────────────┘ └─────────────┘ └─────────────┘
The Key Insight
The data key is the bridge. It's a random symmetric key that:
- Encrypts the master secret (stored in the database as one row)
- Is itself encrypted separately for each admin using their own password
- Never stored in plaintext — only exists in memory after an admin logs in
When Admin A logs in:
- Their password derives an AES key (via Argon2id)
- That AES key decrypts their copy of the data key
- The data key decrypts the master secret
- The master secret is held in memory — the service is now "unlocked"
When Admin B logs in later, the same thing happens with their password and their copy of the data key. Both arrive at the same master secret, but neither ever saw the other's password.
Implementation Walkthrough
Let's build this step by step. I'll use Rust for the backend, but the concepts apply to any language.
Step 1: Key Derivation with Argon2id
When an admin sets their password, we don't just hash it — we derive a 32-byte AES key from it. Argon2id is the gold standard for this: it's memory-hard (resistant to GPU/ASIC attacks) and time-hard.
use argon2::Argon2;
fn derive_key_from_password(password: &str, salt: &[u8; 16]) -> [u8; 32] {
let argon2 = Argon2::new(
argon2::Algorithm::Argon2id,
argon2::Version::V0x13,
argon2::Params::new(
100 * 1024, // 100 MB memory cost
3, // 3 iterations
1, // 1 degree of parallelism
Some(32), // 32-byte output
).unwrap(),
);
let mut derived_key = [0u8; 32];
argon2.hash_password_into(
password.as_bytes(),
salt,
&mut derived_key,
).unwrap();
derived_key
}
Why Argon2id over bcrypt/scrypt? Argon2id won the Password Hashing Competition. The "id" variant combines Argon2i (side-channel resistant) and Argon2d (GPU-resistant) — best of both worlds.
Why 100MB memory cost? It makes brute-force attacks require 100MB per guess. An attacker with 16GB of RAM can only run ~160 parallel attempts. Compare that to billions of SHA-256 hashes per second on a GPU.
Step 2: AES-256-GCM Encryption
Every encryption operation uses AES-256-GCM — authenticated encryption that provides both confidentiality and integrity.
use aes_gcm::{Aes256Gcm, KeyInit, Nonce};
use aes_gcm::aead::Aead;
struct EncryptedBlob {
nonce: String, // 12 bytes, base64-encoded
data: String, // ciphertext + auth tag, base64-encoded
salt: String, // 16 bytes, base64-encoded (only for password-derived keys)
}
fn encrypt(key: &[u8; 32], plaintext: &[u8]) -> EncryptedBlob {
let cipher = Aes256Gcm::new_from_slice(key).unwrap();
let nonce_bytes: [u8; 12] = rand::random();
let nonce = Nonce::from_slice(&nonce_bytes);
let ciphertext = cipher.encrypt(nonce, plaintext).unwrap();
EncryptedBlob {
nonce: base64_encode(&nonce_bytes),
data: base64_encode(&ciphertext),
salt: String::new(),
}
}
Why a random nonce every time? AES-GCM is catastrophically broken if you reuse a nonce with the same key. Random 12-byte nonces give us a comfortable margin — the birthday bound for collision is ~2^48 encryptions, far more than we'll ever do.
Step 3: The Envelope — Putting It Together
Here's how the three layers connect:
// === Layer 1: Encrypt the master secret with the data key ===
fn encrypt_master(data_key: &[u8; 32], master_password: &str) -> EncryptedBlob {
encrypt(data_key, master_password.as_bytes())
}
// === Layer 2: Encrypt the data key with a user's password ===
fn encrypt_data_key_for_user(
user_password: &str,
data_key: &[u8; 32],
) -> EncryptedBlob {
let salt: [u8; 16] = rand::random();
let derived_key = derive_key_from_password(user_password, &salt);
let mut blob = encrypt(&derived_key, data_key);
blob.salt = base64_encode(&salt);
blob
}
// === Unlocking: Reverse the layers ===
fn unlock(
user_password: &str,
user_encrypted_data_key: &EncryptedBlob, // from user's DB row
encrypted_master: &EncryptedBlob, // from global table
) -> String {
// Step 1: Derive AES key from user's password
let salt = base64_decode(&user_encrypted_data_key.salt);
let derived_key = derive_key_from_password(user_password, &salt);
// Step 2: Decrypt the data key
let data_key = decrypt(&derived_key, user_encrypted_data_key);
// Step 3: Decrypt the master secret
decrypt(&data_key, encrypted_master)
}
Step 4: Database Schema
You need surprisingly little storage:
-- One row: the encrypted master secret
CREATE TABLE envelope_master (
id INTEGER PRIMARY KEY CHECK (id = 1),
encrypted_master_blob TEXT NOT NULL, -- JSON {nonce, data, salt}
encrypted_master_recovery TEXT NOT NULL, -- encrypted by recovery key
recovery_key_hash TEXT NOT NULL, -- Argon2id hash for verification
created_at TEXT NOT NULL,
updated_at TEXT NOT NULL
);
-- Each admin gets one extra column
ALTER TABLE users ADD COLUMN encrypted_data_key TEXT;
-- NULL for non-admin roles (they don't have crypto access)
That's it. One global row for the encrypted master secret, and one column per user for their encrypted copy of the data key. Non-admin users (operators, viewers) have NULL in this column — they can authenticate but have zero cryptographic path to the master secret.
Step 5: Backend State Machine
The backend tracks whether it's been "unlocked" by an admin:
enum BackendLockState {
Locked, // No admin has logged in since restart
Unlocked, // Master secret is in memory, service fully operational
ManuallyLocked, // Admin explicitly locked it
}
┌─────────────┐ ┌──────────────────┐
│ Service │────────▶│ LOCKED │◄─────────────────┐
│ Restart │ │ │ │
└─────────────┘ │ No admin has │ ┌────────┴────────┐
│ authenticated │ │ MANUALLY │
└────────┬─────────┘ │ LOCKED │
│ │ │
Admin logs in │ Admin called │
(decrypts envelope) │ /auth/lock │
│ └────────▲────────┘
┌────────▼─────────┐ │
│ UNLOCKED │─────────────────┘
│ │ Manual lock
│ Master secret │
│ held in memory │
└───────────────────┘
When the service restarts, it starts in Locked state. No operations requiring the master secret work until an admin logs in. This is a deliberate tradeoff — the secret only exists in memory when someone has actively authenticated.
Adding and Removing Admins
Adding a New Admin
When an existing admin creates a new admin user:
- The backend already has
data_keyin memory (it's unlocked) - New admin sets their password during onboarding
- Backend encrypts
data_keywith the new admin's password - Stores the encrypted blob in the new admin's DB row
The master secret was never involved. The new admin never sees the data key. They just get their own encrypted copy.
Removing an Admin
- Delete their row from the database (including their
encrypted_data_key) - That's it
Their encrypted blob is gone. They can't decrypt the data key anymore. No re-encryption needed. The data key and master secret remain unchanged — the removed admin's copy is simply destroyed.
Rotating the Master Secret
If you ever need to change the master secret:
- Re-encrypt the new master secret with the existing data key
- Update the single row in
envelope_master - Done — no per-user changes needed
If you need to rotate the data key (e.g., you suspect it was compromised):
- Generate a new data key
- Re-encrypt the master secret with the new data key
- Re-encrypt the new data key for each active admin
- This is O(N) but should be rare
Recovery Key
What if all admins forget their passwords? You need a recovery path.
During initial setup, the system generates a recovery key — a random 256-bit value, base64-encoded. The master secret is also encrypted with this recovery key and stored alongside the normal envelope.
fn generate_recovery_key() -> String {
let key: [u8; 32] = rand::random();
base64_encode(&key)
}
The recovery key is:
- Shown once to the owner during setup (they write it down / store it offline)
- Its hash is stored (Argon2id) so we can verify it without storing it
- Used to decrypt the master secret as a last resort
- After recovery, the admin sets a new password and the envelope is re-wrapped
Think of it as a "break glass in case of emergency" mechanism.
The Role Hierarchy
Not everyone needs crypto access. A clean three-tier model:
| Role | Can Authenticate | Has encrypted_data_key | Can Unlock Service |
|---|---|---|---|
| Admin | Yes | Yes | Yes |
| Operator | Yes | No | No |
| Member | Yes | No | No |
Operators can stop the service, change settings, and perform transfers — but they can't start or restart it (that requires the master secret). Members are read-only. This separation is baked into the cryptography itself, not just access control lists.
Backward-Compatible Migration
If you're adding this to an existing single-admin system, you need a migration path. Here's how we handled it:
Before: Owner's login password = keystore master password (they're the same thing)
After first login post-upgrade:
- Owner logs in with their password (which IS the master password)
- System detects: "envelope not initialized yet"
- Auto-generates
data_keyandrecovery_key - Encrypts master password with data_key → stores in
envelope_master - Encrypts data_key with owner's password → stores in their user row
- Encrypts master password with recovery key → stores in
envelope_master - Shows recovery key to owner (one-time display)
- From now on, the envelope system is active
Zero downtime. No manual migration steps. The owner doesn't even realize the system has changed underneath.
Zeroing Secrets in Memory
One detail that's easy to overlook: secrets in memory should be zeroed when no longer needed. In Rust, the zeroize crate handles this:
use zeroize::{Zeroize, ZeroizeOnDrop};
#[derive(ZeroizeOnDrop)]
struct UnlockedSecrets {
master_password: String,
data_key: [u8; 32],
}
When the backend transitions to Locked state, the UnlockedSecrets struct is dropped and its memory is overwritten with zeros. This prevents the secrets from lingering in memory where they could be extracted via a memory dump.
Lessons Learned
Argon2id is slow by design. Deriving a key takes ~200ms with our parameters. This is a feature, not a bug — but it means you should only do it once per login, not per request. Cache the decrypted data key in memory.
The "first admin must log in" UX is a real tradeoff. After a service restart, nothing works until an admin authenticates. We added a clear "Backend is locked" banner in the UI so operators know what's happening and who to call.
Recovery keys are non-negotiable. Without one, losing all admin passwords means losing the master secret forever. Treat the recovery key like a root CA certificate — offline, secure, tested.
Separation of authentication and authorization is powerful. Operators can log in, see dashboards, stop the service — but the cryptographic design makes it impossible for them to start it. This isn't an access control list that could be misconfigured; it's math.
Migration must be invisible. If upgrading requires manual steps, adoption drops to zero. The auto-migration on first login was the hardest part to implement but the most important for real-world deployment.
Wrapping Up
Envelope encryption isn't new — AWS KMS, Google Cloud KMS, and HashiCorp Vault all use variants of it internally. But you don't need a managed service to implement the pattern. The core idea is simple:
Encrypt the secret with a key. Encrypt that key separately for each person who needs access.
Two layers of encryption. Clean separation of concerns. O(1) secret rotation. O(1) user revocation. And the master secret never leaves memory in plaintext except when it's actively being used.
If you're building any system where multiple people need to unlock a shared secret — whether it's a blockchain keystore, a signing service, or a deployment pipeline — this pattern will serve you well.
Have questions or built something similar? I'd love to hear about it in the comments.
Top comments (0)