What we are testing before choosing how Vault issues AWS credentials, what assumed_role revealed, and where AI-generated tests still needed an engineer.
Replacing long-lived AWS keys sounds straightforward: issue credentials that expire, and stop leaving permanent access in build systems, laptops and old scripts.
We already use HashiCorp Vault to manage secrets. It can issue AWS credentials in four different ways. Before choosing one for wider use, we wanted to understand how each behaves when access is granted, restricted or revoked.
I used an AI-assisted workflow to help with the evaluation. Agents prepared test plans, reviewed one another's work, wrote scripts and turned execution logs into reports. I reviewed the scripts, ran them in a sandbox and checked the conclusions.
All four plans are ready. Two options have been tested. This article covers the comparison we are working through and what the first option, assumed_role, taught us. The AI loop implementation will get its own article.
Four ways to issue access
The goal is to replace keys distributed to people and automated jobs with access that lasts only as long as needed. Shorter lifetimes reduce the window in which a stolen credential can be used. Permissions still matter: a short-lived administrator credential can do plenty of damage before it expires.
Vault's four credential types take different approaches:
| Type | How it works | Main question for our evaluation |
|---|---|---|
assumed_role |
Gets temporary credentials for an AWS role, which defines the available permissions. | How do duration limits, audit trails and early revocation behave? |
iam_user |
Creates an IAM user and key, then deletes the user when its Vault lease expires. | How reliable are creation and cleanup, and how much authority must Vault hold? |
federation_token |
Gets temporary credentials limited by both the issuing user's permissions and additional session policies. | How do those policies combine to limit access? |
session_token |
Gets temporary credentials based on the issuing IAM user's permissions, without a separate Vault role policy to narrow them. | Does this give callers only the permissions they need? |
These are documented differences, not four completed test results. In particular, the iam_user key does not expire by itself: Vault has to remove it as part of deleting the user. With the STS options, AWS enforces expiry. With iam_user, we also depend on Vault successfully cleaning up.
We gave each option a ticket and a sandbox test. The practical questions were how quickly access becomes usable, what callers can do, whether we can trace their actions, and how access ends. We also wanted to understand the consequences if Vault's issuing credentials leaked.
Where the AI loops helped
The workflow produced a plan for each ticket, including possible failures, then had separate AI reviewers check it. They wrote scripts for setup, testing and cleanup, and produced reports from the logs after I ran them.
The agent did not receive the AWS keys or Vault token. I supplied credentials to the scripts when running them, and every live action in this evaluation came from a script I ran. That left me responsible for reviewing what those scripts would do.
This gave us a repeatable way to evaluate the options. It also meant fixes from the first test could be carried into later scripts. The separate reviewers were useful, although one invalid test still made it through.
First result: assumed_role
For this test, Vault used an IAM user allowed to assume roles matching a particular name prefix. Each role defined what the issued credentials could do.
That still leaves an underlying long-lived key in Vault in our test setup. We are evaluating how to replace keys handed out to people and jobs, rather than claiming to eliminate every static key at once.
Access worked on the first attempt
In our test, the first request succeeded one second after Vault issued the credentials, with no retries. That matters for automated jobs that request access and immediately use it.
Our session name included the Vault login and role. We saw it in CloudTrail, AWS's audit log, so we could connect the activity to the requester. That is more useful than a shared static key which identifies an IAM user but often says little about which person or job used it. The naming template needs to fit within the 64-character session-name limit.
The permission checks also behaved as intended. The test role could list S3 buckets; listing EC2 regions was denied. The issuing user could not assume a role outside its allowed prefix and could not list IAM users.
Those checks supported the intended restrictions. A leaked issuing key could still request fresh sessions for the roles it was allowed to assume, so protecting it remains important.
Two duration limits need to agree
AWS rejected an AssumeRole session shorter than 15 minutes. At the upper end, Vault and AWS handled excessive requests differently:
- Vault capped a request at its configured
max_sts_ttl. - AWS rejected a request above the role's maximum session duration.
If Vault allows two hours but the AWS role allows only one, a two-hour request fails. These settings need to be configured together so callers get predictable behaviour.
Revoking in Vault did not stop access in AWS
This was the most important operational finding. Revoking the Vault lease did not invalidate the issued role credentials. Without an AWS-side permission change, they remained usable until their AWS expiry.
AWS provides a way to block older role sessions using a deny policy with a cutoff time. Its standard “revoke active sessions” action uses that approach. It also affects legitimate users of the same role whose sessions started before the cutoff, so they need fresh credentials.
That is something to establish before an incident. “Revoke access” can mean different things in Vault and AWS, and the emergency procedure can interrupt other work.
The test that answered the wrong question
We also wanted to measure how quickly the AWS-side deny policy took effect. The generated report said it had “no effect within two minutes”.
Reading the log, I found that the script was checking access with sts:GetCallerIdentity, essentially asking AWS “who am I?”. AWS allows this request even when an identity policy explicitly denies it. It could not tell us whether the session had lost permission to do useful work.
The conclusion became “not measured”. I asked the agent to replace the check with an action that could actually be denied. The script has been corrected, but we still need a valid timing result.
This showed a weakness in the review process. The plan, script and report all relied on the same wrong assumption. I only caught it when I checked what the API call actually proved.
What still needed engineering work
The first run stopped because my Vault admin policy lacked access to the new test paths. I cleaned up using the rollback steps, then made about eight fixes to the script: safer cleanup, stricter checks and a permissions check before creating resources. I updated the Vault policy through our normal code review process.
The second run failed at the revoke step because of another missing permission. This time cleanup ran automatically.
The workflow carried these fixes into the later scripts. That was useful, but the work still needed someone who understood the permissions, checked cleanup and could challenge a report's conclusion.
For context, the workflow's own estimate of its first week's model usage was about $14, excluding my interactive AI sessions and my time. My involvement was very roughly one to two hours per ticket. These are estimates, not a measured comparison with doing the work manually.
What we still need before choosing
The iam_user test has also run; its results will be covered in the follow-up. It requires Vault to create and manage IAM users and keys, which makes the issuing identity's permissions and reliable cleanup especially important.
The federation_token and session_token plans are reviewed but have not run yet. We still need those results before choosing an approach for wider use.
We have useful results for assumed_role, but not enough evidence to choose among all four options yet. The AI workflow helped prepare and document the tests. My job was still to check that those tests answered the questions we needed answered.
References: Vault AWS secrets engine, Vault AWS API, AWS AssumeRole, revoking role session permissions, and GetCallerIdentity.
Top comments (0)