How to keep your database passwords and API keys safe, and why rotating them doesn't break your app the way you think it will
If you are new to AWS, there is a good chance you have done something like this at least once. You needed your app to connect to a database, so you took the database password and pasted it straight into your code, or into a Lambda environment variable, and moved on. It worked, so why worry?
The problem is that a password sitting in plain view is a password waiting to leak. In this tutorial we will walk through AWS Secrets Manager, a service built to store exactly these kinds of sensitive values safely. We will cover what it is, why environment variables are a weak place to keep secrets, what a KMS key actually does, and the one topic that confuses almost every beginner: automatic rotation. By the end you will understand why rotation does not break your application, even though it changes your password while the app is running.
Let us start from the beginning.
Jargon check before we go further: a secret just means any sensitive string your app needs but should never be public. Think database passwords, API keys, and access tokens. An API key is a long secret string that proves your app is allowed to use some service. A token is similar, usually a temporary one.
What Is AWS Secrets Manager?
AWS Secrets Manager is an encrypted vault for your secrets. Instead of writing your database password directly into your code, you store it inside Secrets Manager, and your app asks for it at runtime (meaning while the app is actually running, not while you are writing it).
A simple way to picture it is a hotel room safe. You do not tape your passport to the wall. You lock it in the safe, and only someone with the right key can open it. The hotel also keeps a log of every time the safe is opened. Secrets Manager works the same way.
It gives you three main benefits.
The first is encryption at rest. "At rest" means while the data is sitting in storage, as opposed to while it is travelling over the network. Your secret is scrambled so that even someone who somehow gets to the raw storage cannot read it.
The second is IAM controlled access. IAM stands for Identity and Access Management, which is the AWS system that decides who is allowed to do what. With Secrets Manager you can say "only my application is allowed to read this database password" and nothing else can touch it.
The third is automatic rotation, which means AWS can change your password on a schedule for you. This is the feature we will spend the most time on, because it is the most useful and the most misunderstood.
Why Not Just Use a Lambda Environment Variable?
An environment variable is a setting you attach to your app from the outside, so the value lives next to the app rather than inside your code. In AWS Lambda (the service that runs your backend code without you managing a server) you can set these in the function configuration. It works, and plenty of people do it. So what is wrong with it for secrets?
There are four real problems.
First, the value sits in plain text inside your function configuration. Anyone who can open the AWS console and view your Lambda can simply read the password. There is no lock on it.
Second, the value tends to leak into places you did not expect. If you manage your infrastructure with a tool like Terraform, the password ends up written into your Terraform state file, which is the record Terraform keeps of everything it created. It can also show up in deployment logs. So one password quietly becomes several copies scattered around.
Third, changing the value means redeploying or updating the function. There is no clean way to swap it on the fly.
Fourth, there is no audit trail. You cannot answer the question "who read this secret and when," because nothing was recorded.
Environment variables are perfectly fine for values that are not sensitive, like which region your app runs in or a feature toggle. They are just a weak place to keep real secrets.
What Is a KMS Key?
When we said Secrets Manager encrypts your data, something has to do the actual encrypting. That something is a KMS key.
KMS stands for Key Management Service. A KMS key is the cryptographic key used to scramble (encrypt) and unscramble (decrypt) your secret. Encryption is just the process of turning readable text into unreadable text so that only someone with the key can read it again.
Here is the clever part. You never see or hold the raw key yourself. It lives inside tamper resistant hardware, meaning special equipment built so the key cannot be copied out or stolen. You simply ask KMS "please decrypt this for me," and KMS checks your IAM permission before doing it.
Think of KMS as the master key to a building, locked inside a vault that you personally cannot open. You do not get to carry the master key around. Instead you hand your locked box to the security desk and say "open this please," and they check your badge first. Because the key never leaves the vault, nobody can walk off with it.
When Secrets Manager stores your password, it quietly uses a KMS key to encrypt it. AWS gives you a default key for free, so as a beginner you usually do not need to create your own.
Automatic Rotation: The Part Everyone Misunderstands
Rotation means automatically changing a password to a new one on a schedule. Here is the worry almost every beginner has, and it is a reasonable one:
"If Secrets Manager changes my password, but my database still has the old one, won't my app stop being able to connect?"
The answer is no, and understanding why is the heart of this whole topic.
Rotation does not change the password in only one place. It changes it in both the database and the secret, as a single coordinated operation. Here is the exact sequence the rotation process runs.
- It generates a brand new random password.
- It connects to the database using the current password (which still works at this point) and runs the command
ALTER USER, which is the SQL instruction that sets a new password on the database account. Now the database expects the new password. - It stores that new password back into Secrets Manager as the current value.
- It connects once more using the new value to confirm everything works. Because both sides are updated together, they are never out of sync in a way that locks you out.
But there is a second condition you must meet for this to stay smooth. Your app has to fetch the secret fresh from Secrets Manager at runtime, rather than holding its own permanent copy. If your code reads the current value when it needs it, then after rotation it simply reads the new value next time and carries on.
There is one trap here worth naming. To avoid asking Secrets Manager for the password on every single request (which costs a little money and adds delay), developers often cache it, meaning they keep a copy in memory for reuse. That is good practice, but if you cache the password forever and never refresh it, then after rotation your app is holding a stale password and will fail to connect. The fix is a cache with a TTL, which stands for Time To Live, a setting that tells the cache to throw the value away and fetch a fresh one after a few minutes.
So the rule is simple. Rotation does not break your app, as long as your app reads the secret fresh and does not cling to an old cached copy.
Versioning: Why the Old Password Sticks Around for a Moment
A secret in Secrets Manager does not hold just one value. It holds versions, and each version carries a label that tells you what it is for. Three labels matter.
AWSCURRENT is the value you get by default when you ask for the secret. This is the live, in use password.
AWSPREVIOUS is the value from just before the last change. Secrets Manager keeps it valid for a short window on purpose. Picture rotation happening at the exact moment a request is already in flight, meaning a request that started a split second before the change. That request may have grabbed the old value already. Keeping AWSPREVIOUS alive briefly lets that in flight request finish successfully instead of erroring out. It smooths over the handover.
AWSPENDING is the new value during rotation, after it has been created but before it has been promoted to current. It is the password in waiting.
You rarely need to think about these directly, because asking for the secret gives you AWSCURRENT automatically. But knowing the labels exist helps you understand why rotation is so smooth and why the brief overlap does not cause failures.
Cross Region Replication: A Feature for Later
By default, a secret lives in a single AWS region. A region is a geographic location where AWS runs its data centers, such as Mumbai (named ap-south-1) or Singapore.
Cross region replication keeps an automatically synced copy of your secret in other regions you choose. If you store a secret in Mumbai and replicate it to Singapore, then any change in Mumbai is copied across to the Singapore version on its own.
You would want this in two situations. One is disaster recovery, so that if an entire region has an outage, an app running in your backup region can still read the secret locally. The other is when your app runs in several regions at once and you want each one to read the secret nearby instead of reaching across the world every time.
As a beginner running a single region project, you almost certainly do not need this yet. File it away as a "when I grow into multiple regions" feature.
Conclusion
If you take away one idea from this tutorial, let it be this. Secrets stay in sync because rotation changes both sides at once, the database and the stored secret, in a single coordinated operation. And your application stays connected because it reads the current value fresh from Secrets Manager whenever it needs it, rather than holding a hardcoded copy that goes stale.
Everything else builds on that. Environment variables are weak for secrets because they sit in plain text and leak into your state files and logs. A KMS key does the encrypting behind the scenes without ever letting you touch the raw key. Versioning with AWSCURRENT and AWSPREVIOUS keeps a brief safety net so in flight requests survive a rotation. And cross region replication is there for the day you outgrow a single region.
So the next time you are tempted to paste a password into an environment variable, reach for Secrets Manager instead. Your future self, and anyone who inherits your code, will thank you.
Get in Touch
Found this helpful, or stuck on something? I would love to hear from you. Reach out at khantanseer43@gmail.com and I will do my best to help.
Top comments (0)