DEV Community: Alex

Give Your Auditor AWS Access — Without Sharing a Single Credential

Alex — Fri, 10 Jul 2026 19:20:42 +0000

If your auditor asks for that, that should be the first finding. 🚩

An external auditor needs access to your AWS accounts, and the first instinct in most companies is one of two things: create an IAM user and share the access keys, or provision the auditor a user in your identity provider so they can log into the console.

Neither is necessary. An auditor doesn’t need a login in your organization at all — API access is enough. And AWS has had a mechanism for exactly this situation for over a decade. It’s the same one AWS partners and every third-party monitoring tool use:

A cross-account IAM role with a trust relationship. No credentials ever change hands.

In this article I’ll walk through how the pattern works and share two CloudFormation templates I built — one for a single AWS account, one that covers an entire AWS Organization. You can deploy either in about five minutes. Both templates are also on GitHub: https://github.com/up-cdk/aws-audit-access

Why not just create a user?

Because every credential you create is a liability you now own:

Access keys are long-lived. They work until someone remembers to delete them. Every audit I’ve read about that went wrong involved keys that outlived the engagement.
Passwords get shared insecurely. Email, Slack, a text message — the moment credentials travel, you’ve lost control of where they end up.
IdP users pollute your directory. An external party in your identity provider inherits whatever group memberships and app assignments someone clicks together, and deprovisioning is a manual step someone has to remember.

The pattern: cross-account role + External ID

The idea is simple. Instead of giving the auditor your credentials, you create a role in your account that the auditor’s own AWS account is allowed to assume.

Four pieces make it safe:

The trust policy names exactly one AWS account — the auditor’s. Not a user, not a password: an account-to-account trust. The auditor authenticates in their account with their credentials, then calls STS AssumeRole to get temporary credentials for the role in yours.
An External ID guards the trust. This is a shared secret agreed between you and the auditor, set as a condition on the trust policy. Without presenting it, the role cannot be assumed — even by the trusted account. This protects against the confused deputy problem: if the auditor works with many clients, a malicious client can’t trick the auditor’s tooling into accessing your account, because they don’t know your External ID.
All access is temporary. STS sessions expire (1 hour by default in these templates, configurable). There is no long-lived credential anywhere in the setup.
Everything is logged. Every AssumeRole call and every API call the auditor makes lands in your CloudTrail, attributable to the audit role’s session. You can see exactly what was looked at, and when.

And the permissions? Read-only, and scoped for the job: AWS’s managed ReadOnlyAccess and SecurityAudit policies, optionally AWSBillingReadOnlyAccess for cost review, plus a small supplementary policy for audit-relevant services the managed policies miss (GuardDuty findings, Security Hub, Cost Explorer, Trusted Advisor, and a few others).

Option 1: single account

Deploy this template in the account being audited. Two parameters matter: the auditor’s 12-digit AWS account ID, and the External ID they give you.

Template: aws-audit-readonly-role.yaml in the GitHub repo — https://github.com/up-cdk/aws-audit-access

Deploy it from the console (CloudFormation → Create stack → Upload template), or from the CLI:

aws cloudformation deploy --template-file aws-audit-readonly-role.yaml --stack-name external-audit-access --capabilities CAPABILITY_NAMED_IAM  --parameter-overrides AuditorAWSAccountId=111122223333 ExternalId=the-external-id-from-your-auditor

Then send the auditor the role ARN from the stack outputs. That’s it — that’s the entire exchange. An ARN is not a secret; combined with the External ID (which the auditor already has, since they generated it) and the trust policy, it’s everything they need and nothing more.

Option 2: the entire AWS Organization

Auditing one account is rare — most audits cover an Organization with anywhere from a handful to hundreds of member accounts. Creating the role by hand in each one doesn’t scale.

The second template solves this with a service-managed CloudFormation StackSet. You deploy one stack in the management account, and it:

creates the audit role in the management account itself,
creates a StackSet that rolls the same role out to every member account under the OUs you target (pass your root ID, e.g. r-abcd, to cover the whole organization),
optionally auto-deploys to any new account added to those OUs later — no gaps if accounts are created mid-audit,
and tears everything down centrally when you delete the stack.

One prerequisite: service-managed StackSets require trusted access between AWS Organizations and CloudFormation to be activated (CloudFormation console → StackSets → “Activate trusted access”, a one-time click in the management account).

Template: aws-audit-organization-stackset.yaml in the same repo. The deploy command:

aws cloudformation deploy --template-file aws-audit-organization-stackset.yaml --stack-name external-audit-access-org --capabilities CAPABILITY_NAMED_IAM --parameter-overrides AuditorAWSAccountId=111122223333 ExternalId=the-external-id-from-your-auditor TargetOrganizationalUnitIds=r-abcd DeploymentRegions=us-east-1

IAM is a global service, so one deployment region is enough for the stack instances.

After deployment, every in-scope account contains a role with the same name, so the auditor can iterate over account IDs with a single ARN pattern:

arn:aws:iam:::role/ExternalReadOnlyAuditRole

The auditor’s side

From their own account, the auditor assumes the role:

aws sts assume-role --role-arn arn:aws:iam::444455556666:role/ExternalReadOnlyAuditRole --role-session-name aws-audit --external-id THE_EXTERNAL_ID

For an organization audit, the auditor loops the same profile pattern over the account list — which they can fetch with organizations:ListAccounts, included in the role’s permissions.

Ending the audit

This is my favorite part, because it’s one line:

aws cloudformation delete-stack --stack-name external-audit-access

Delete the stack (or the org stack, which removes the StackSet instances from every member account). Access is gone. Nothing to rotate, nothing to deprovision, no forgotten IAM user with active keys waiting for the next auditor to find.

The way you grant audit access says a lot about how you manage access everywhere else. If your next audit starts with “please create a user for us” — send them this article instead.

I tried Karpathy's LLM Wiki pattern on my AWS consulting work — here's what changed

Alex — Tue, 14 Apr 2026 00:22:22 +0000

I tried Karpathy's LLM Wiki principle on my own AWS consulting work a few weeks ago, and I can't go back.

If you missed it, Andrej Karpathy published a short gist called LLM Wiki: A Pattern for Persistent Knowledge Bases. His thesis: instead of using RAG to re-derive answers from raw sources on every query, let the LLM maintain a persistent, compounding wiki. Humans curate sources and ask questions. The model handles the bookkeeping — summarizing, cross-linking, keeping pages consistent over time.

So I tried it. I rebuilt my entire AWS consulting knowledge base around Karpathy's three-layer pattern. This post is what I actually did, what the folder looks like now, the frontmatter schema that makes it queryable by Claude Code skills, and the three rules I had to enforce to stop it from rotting.

At the bottom there's a copy-pasteable Claude Code prompt that scaffolds the same structure in an empty directory — if you want to try it yourself.

Why I tried this on AWS consulting specifically

Every engagement I run has the same four information streams, and all four used to decay fast:

Client-specific context — their account structure, compliance posture, team, kickoff constraints.
Cross-project AWS patterns — cross-account alarm routing, IAM boundaries, cost allocation.
Decisions — why I picked Aurora over RDS for this client, what tradeoff I accepted.
Gotchas — what broke, in what AWS service, under what conditions.

Before Karpathy's pattern, all four lived, well not in the same place let's say. Every new engagement brought the same conclusion that I was the bottleneck :).

Now the LLM compiles it once and keeps it current for me. I'm not the bottleneck anymore.

The three layers, as I actually set them up

Layer 1 — Raw sources (immutable)

./raw/
├── assets/                      # Downloaded images, screenshots
├── aws-reinvent-2024-obs.md     # Clipped article
├── client-kickoff-2026-03.md    # Meeting transcript
├── datadog-quote-client-a.pdf   # Vendor docs
└── aws-bill-client-b-q1.csv     # Cost exports

Raw never changes. If the wiki layer gets corrupted or I change conventions, I can re-derive from raw. That immutability is what gave me the confidence to let Claude aggressively refactor the wiki.

Layer 2 — Wiki (LLM-maintained markdown)

./wiki/
├── projects/
│   ├── client-a/
│   │   ├── _status.md             # Dashboard — I start here every session
│   │   ├── client-a-overview.md   # Architecture + key decisions
│   │   ├── decisions/             # ADR-style, one file per decision
│   │   ├── workflows/             # Automation-ready procedures
│   │   └── gotchas/               # What broke and why
│   └── client-b/
├── sa/                            # Solutions architecture patterns (cross-client)
├── aws/                           # AWS service patterns + gotchas
└── standards/                     # CDK conventions, tagging, account standards

One vault, per-client folders. Cross-client patterns live in sa/, aws/, standards/ — my opinion library. Client folders stay isolated so I never accidentally leak Client A's IAM choices into a Client B audit.

Layer 3 — `CLAUDE.md` (the schema)

Karpathy calls this the "config for how the LLM operates on the wiki." For me, it's a single file at the vault root that defines the directory structure, the frontmatter schema, what belongs in the wiki and what doesn't, and the workflows I want Claude to follow (ingest a source, update _status.md, lint for orphans).

Change CLAUDE.md and you change how every future Claude Code session operates on the vault. It's the load-bearing file.

The frontmatter schema I landed on

Every page has YAML frontmatter. Not for me to read — for skills to query:

---
title: "Cross-account CloudWatch alarm routing"
type: pattern
tags: [aws, cloudwatch, cross-account, observability]
sources: [raw/aws-reinvent-2024-obs.md, project: client-a]
created: 2026-03-02
updated: 2026-04-11
---

sources is non-negotiable for me. It keeps the agent honest when it synthesizes — every claim in the wiki traces back to something in raw.

The first skill I wrote that made me a believer

Here's the one that convinced me this was worth the effort: an architecture-review skill I can run when a client proposes a change.

---
name: architecture-review
description: Reviews a proposed architecture change against client constraints and cross-client patterns
---

Inputs:
1. Client name (must match a folder in wiki/projects/)
2. Proposed change (free text or linked diagram)

Steps:
1. Read wiki/projects/<client>/_status.md for current phase and constraints
2. Read wiki/projects/<client>/<client>-overview.md for architecture baseline
3. Query wiki/sa/ and wiki/aws/ for pages where tags intersect with the change
4. Query wiki/projects/<client>/gotchas/ for anything relevant
5. Query wiki/projects/<client>/decisions/ for prior decisions that constrain this change
6. Produce a review document with:
   - Compatibility with existing architecture
   - Relevant cross-client patterns that apply
   - Prior decisions that would be affected
   - Gotchas that could bite
   - Recommended next steps

Output: wiki/projects/<client>/decisions/<YYYY-MM-DD>-<slug>.md (draft for human review)

There's no magic prompt here. The whole point is that the skill is running against structured, maintained context — not a blank LLM. The quality of the output is about the quality of the wiki, not the prompt.

The three rules I had to enforce

Karpathy warns that maintenance is what kills human wikis. Even with an LLM doing the work, I found three rules that actually matter in practice:

Frontmatter or it didn't happen. No frontmatter = skills can't index it. I make Claude add frontmatter before content on every new page.
_status.md updates at session end, not start. I set up a Claude Code Stop hook that blocks me from ending a session until the dashboard moves forward. Without the hook, dashboards went stale inside a week.
One page, one concept. Gotchas get their own pages, not subheadings under a shared "troubleshooting.md". The retrieval unit is a page — keep it small enough that multiple fit in context without bleed.

What actually changed in my work

Three concrete things shifted after a few weeks:

Architecture designs are regenerated, not rewritten. A client constraint changes → I update the relevant page → rerun the design skill → new design doc.
Audits are reproducible. Same vault + same skill on the same day = same findings. I couldn't get that property out of ad-hoc Claude chats no matter how careful I was with prompts.
Handover is the deliverable. At the end of an engagement, the client gets the CDK and their vault folder. The vault is the documentation.

The output layer

Everything Claude generates in my setup ends up as CDK. I open-sourced the constructs at up-cdk — compliance-aligned building blocks for S3, VPC, pipelines, so every stack is compliant by default. Wiki + skills + opinionated output library is the combination that finally made the setup reproducible across clients.

If you want to try it — the bootstrap prompt

The fastest way to see if this pattern works for you is to try it. Here's the Claude Code prompt I use to scaffold the whole structure in an empty directory. No scripts, no deps, just markdown and conventions.

You are setting up a new LLM Wiki following Andrej Karpathy's pattern
(https://gist.github.com/karpathy/442a6bf555914893e9891c11519de94f),
adapted for AWS consulting / platform engineering work.

The wiki has three layers:
- `raw/` — immutable source material (meeting notes, clipped articles, bills, screenshots)
- `wiki/` — LLM-maintained markdown synthesized from raw + conversation
- `CLAUDE.md` at the root — the schema document that tells you how to operate on the wiki

Create the following structure in the current directory. For every file below,
create it with the frontmatter and seed content described. Do not skip any
file. Do not invent additional files.

## Directory tree

./
├── CLAUDE.md
├── index.md
├── log.md
├── raw/
│   ├── assets/.gitkeep
│   └── README.md
├── templates/
│   ├── project-status.md
│   ├── page.md
│   └── workflow.md
└── wiki/
    ├── projects/
    │   └── _example-client/
    │       ├── _status.md
    │       ├── _example-client-overview.md
    │       ├── decisions/.gitkeep
    │       ├── workflows/.gitkeep
    │       ├── gotchas/.gitkeep
    │       └── diagrams/.gitkeep
    ├── sa/.gitkeep
    ├── aws/.gitkeep
    └── standards/.gitkeep

## CLAUDE.md (the schema document)

Write a root CLAUDE.md that defines:
- Purpose: personal AWS knowledge base following Karpathy's LLM Wiki pattern.
- The three layers: raw/ (immutable), wiki/ (LLM-generated), CLAUDE.md (schema).
- What belongs in the wiki: durable knowledge (gotchas, decisions, patterns,
  strategy, workflows, status dashboards). What does NOT belong: raw doc
  summaries, git state, ephemeral task context.
- Page format — every wiki page must have YAML frontmatter with fields:
  title, type (pattern | gotcha | decision | concept | process | status |
  workflow), tags, sources, created, updated.
- File naming — kebab-case. Overview files: <project>-overview.md. Status
  files: _status.md (underscore prefix sorts to top).
- Cross-references — [[filename]] Obsidian-style wiki links.
- Every page ends with a ## Related section.
- Workflows (document all four):
  1. Quick context recovery at session start — read
     wiki/projects/<project>/_status.md first.
  2. Update _status.md at session end — refresh Last/Next session lines,
     append one row to Session History (keep last 5), update Recent Decisions
     and Open Questions, set updated: to today.
  3. Ingest a raw source — read thoroughly, write/update pages in the right
     wiki/ subdirectory, cross-link, update index.md, append to log.md.
  4. Lint — find orphan pages, stale content, missing back-links.
- Common tags taxonomy — starter list for AWS, SA, meta tags.
- Persistence layer rules — wiki is the single source of truth for durable
  knowledge.

## index.md
A one-page catalog. Headers for each top-level wiki directory (## Projects,
## Solutions Architecture, ## AWS, ## Standards), each with "No pages yet."

## log.md
Append-only activity log. Seed with today's date and "Initialized wiki
following Karpathy LLM Wiki pattern."

## templates/project-status.md
Template for _status.md files. type: status. Sections: Phase, Last session,
Next session, Current State, Recent Decisions, Open Questions, Session
History (Date | Summary table), Quick Links.

## templates/page.md
Generic page template. Frontmatter + ## Key Points / ## Details / ## Related
/ ## Sources.

## templates/workflow.md
Automation-ready procedure template. type: workflow. Sections: Goal,
Prerequisites, Steps (numbered, imperative), Verification, Rollback (if
applicable), Related.

## wiki/projects/_example-client/_status.md
Realistic example status dashboard. Phase: active. Plausible placeholder
content. Marked as example.

## wiki/projects/_example-client/_example-client-overview.md
Example architecture overview. Frontmatter + 3-sentence overview + Key
Points + Architecture + Recent Decisions + Related.

## raw/README.md
One-page explanation of what goes in raw/ and the immutability rule: never
edit raw files after adding them.

---

After creating everything, print a summary (file count + tree) and next
steps: (1) add your first raw source to raw/, (2) create your first real
project folder under wiki/projects/, (3) commit to git.

Do not add files beyond what's specified. Do not add a root README.md.
Do not add a LICENSE.

Run it from an empty directory, then git init, then add your first raw source. I had a working LLM Wiki in under two minutes.

The original gist is short and worth reading: Karpathy — LLM Wiki.

If you try it, drop a comment with how you ended up shaping it. I'd genuinely like to compare notes.