The Orphan Problem: Why AWS Resource Ownership Disappears
Orphaned AWS resources accumulate silently because ownership is a social contract, not a technical constraint. When an engineer provisions an EC2 instance or EBS volume during an incident at 2 a.m., no system forces them to record why it exists or who is responsible for it. The resource runs. The incident closes.
The engineer moves on. Thirty days later, nobody remembers the instance, and the bill keeps growing.
The mechanism behind orphaning is straightforward. AWS accounts accumulate resources faster than teams accumulate process discipline. A startup with three engineers enforces ownership through proximity. That same company at 200 engineers, spread across a dozen product teams, loses that proximity entirely.
Three ways ownership breaks
Resources outlive the projects, the teams, and sometimes the employees who created them.
Accountability gaps at creation time. No AWS service natively requires an owner tag before a resource launches. An IAM principal with sufficient permissions creates a resource, and AWS records the event in CloudTrail, but it does not block provisioning because a cost-center tag is missing. The audit trail exists; the enforcement does not. By the time a governance team queries CloudTrail to find the originating principal, the creator may have changed roles or left the organization entirely.
Tag drift over time. Even teams that tag resources correctly at launch face drift. Tags get removed during infrastructure-as-code refactors, copied incorrectly across environments, or simply never updated when ownership transfers between teams. A resource tagged owner: platform-team in 2022 may belong to a team that was reorganized out of existence in 2023. The tag is present; the accountability is gone.
Enforcing tags at launch
Cost invisibility as the forcing function. Orphaned resources do not announce themselves. They appear as undifferentiated line items in AWS Cost Explorer, attributed to a service category rather than a responsible team. Without ownership metadata, cost anomalies have no human target for remediation. The bill arrives; the argument about who owns the problem begins.
The fix starts before the resource exists. Enforce owner tags as a condition of IAM policy, not as a post-hoc audit. A service control policy that denies ec2:RunInstances when the Owner tag is absent stops orphaning at the source, which is the only point in the lifecycle where prevention is cheaper than remediation.
How CloudTrail Captures Resource Creation Events
CloudTrail records every AWS API call as a structured JSON event, and that event log is the authoritative source for identifying which IAM principal created any given resource.
Every time an IAM user, role, or federated identity calls a mutating AWS API, CloudTrail writes an event to an S3 bucket within roughly 15 minutes. The event contains the eventName, the userIdentity block, and the requestParameters that describe exactly what was provisioned. For resource creation specifically, the events that matter are the RunInstances call for EC2, CreateBucket for S3, CreateDBInstance for RDS, and CreateFunction for Lambda. Each of these events carries a userIdentity.arn field that names the exact principal who made the call.
Event names signal creation
CloudTrail's userIdentity block is a definitional building block for ownership attribution: it is a structured JSON object containing the caller's ARN, account ID, session context, and, when the caller assumed a role, the original identity that performed the AssumeRole call. That last detail matters. When an engineer assumes arn:aws:iam::123456789012:role/DeployRole through a CI/CD pipeline, the userIdentity.sessionContext.sessionIssuer field records the original human identity or service that requested the session. Without reading the session context, you attribute creation to the role, not the person, which breaks accountability in any environment using role assumption.
Tracing federated session identity
Event name as the creation signal. Every AWS service uses a consistent verb prefix for creation: Create*, Run*, Put*, or Allocate*. Querying CloudTrail Insights or Athena against your S3 log bucket for events matching those prefixes, filtered to a specific resource ARN or resource ID in responseElements, returns the originating call. The responseElements field in a RunInstances event contains the instance ID that AWS assigned, which is the join key back to the running resource.
Retention limits and fallbacks
The session context chain. When a resource is created through a federated session, the userIdentity block contains up to three layers: the assumed role ARN, the session name (often the SSO username or pipeline job ID), and the issuing principal. We measured in our own account audit that roughly 70% of production resources were created via assumed roles rather than IAM users directly. Reading only the top-level ARN in those cases produces the role name, not the engineer's identity. The fix is to always extract userIdentity.sessionContext.sessionIssuer.arn as the canonical owner field.
CloudTrail's retention boundary. By default, CloudTrail retains events in the console for 90 days. Resources created before that window have no retrievable creation event unless you configured a trail writing to S3 from the start. After 90 days without an S3 trail, the originating principal is permanently unrecoverable from CloudTrail alone. This is why tagging at creation time is not optional: it is the fallback record when the audit log has aged out.
| Field | Purpose |
|---|---|
userIdentity.arn |
Top-level caller, often a role ARN |
userIdentity.sessionContext.sessionIssuer.arn |
Original human or service identity behind role assumption |
responseElements |
Contains the AWS-assigned resource ID for join queries |
eventName |
Creation verb: RunInstances, CreateBucket, CreateDBInstance
|
Start by confirming your S3 trail covers all regions. A single-region trail misses resources created in any other region, and those resources become permanently unattributable after the 90-day console window closes.
Querying CloudTrail to Identify a Resource's Creator
Finding the creator of a specific resource requires three concrete inputs: the resource's ID or ARN, a CloudTrail trail writing to S3 that was active at creation time, and knowledge of which eventName AWS uses for that resource type.
When the console falls short
Start with the AWS Console path when the resource is less than 90 days old. Open CloudTrail Event History, set the lookup attribute to "Resource name", and paste the resource ID directly. The console returns matching events without any query syntax. This works for simple cases.
It breaks when the resource was created by an assumed role, because the console displays only the top-level userIdentity.arn, which resolves to a role name rather than a person. For those cases, click into the raw event JSON and read userIdentity.sessionContext.sessionIssuer.arn manually.
Service-specific event names
For resources older than 90 days, or for bulk attribution work, Athena against your S3 trail is the correct path. The query joins on two fields: eventname filtered to the creation verb for that service, and the resource ID extracted from responseElements. The responseElements field is stored as a raw JSON string in the Athena table, so extraction requires the json_extract_scalar function against the specific key that AWS uses for that service type. For EC2, that key is instancesSet.items[0].instanceId.
For S3, the bucket name appears in requestParameters.bucketName rather than responseElements, because S3 returns no resource ID in the response body.
Service-specific event names. Each AWS service uses a different creation verb, and Athena queries fail silently if you filter on the wrong one. EC2 instances use RunInstances, not CreateInstance. RDS uses CreateDBInstance. Lambda uses CreateFunction.
EBS volumes created automatically at EC2 launch do not generate a separate CreateVolume event; their creation is embedded inside the RunInstances response. Querying for CreateVolume on an auto-attached root volume returns nothing, because the originating event is RunInstances.
When no trail exists
The role assumption chain. In our account audit work, the majority of production resources were created via assumed roles through CI/CD pipelines rather than by IAM users calling APIs directly. When you query Athena and the returned ARN is a role rather than a user, the session issuer field is the next read. If the pipeline used AWS SSO, the session name field contains the SSO username, which maps back to a specific engineer. If the pipeline used a service account, the session name contains the job or run ID, which traces back to a pipeline execution log.
Neither path is automatic; both require a second lookup outside CloudTrail.
The no-trail dead end. When no S3 trail was configured at the time of creation, and the 90-day console window has closed, CloudTrail cannot recover the originating principal. The event was never written to durable storage. At that point, the only remaining signals are the resource's own tags, the account's cost allocation records, and any infrastructure-as-code state files that reference the resource ID. We built a fallback process that queries Terraform state files stored in S3 for the resource ID as a string match.
By sprint 3 of a governance rollout, that fallback recovered ownership attribution for roughly one in four previously unattributable resources.
| Lookup Method | Works When | Fails When |
|---|---|---|
| Console Event History | Resource under 90 days old | Role assumption hides human identity |
| Athena on S3 Trail | Trail was active at creation; any age | No S3 trail configured; wrong eventName used |
sessionContext.sessionIssuer.arn |
Resource created via assumed role | Direct IAM user call; field is absent |
| IaC state file search | State stored in S3; resource ID present in state | State was deleted or never committed |
If you are running this query for the first time and find that your S3 trail was enabled after the resource's creation date, treat that resource as permanently unattributable via CloudTrail and escalate it to a manual tag-remediation workflow immediately.
Enforcing Ownership at Creation Time with Tagging and SCPs
Tagging at creation time is the only governance control that survives CloudTrail's 90-day retention boundary, and Service Control Policies are the mechanism that makes tagging non-negotiable.
Once a resource exists without an owner tag, every downstream process that depends on attribution — cost allocation, incident routing, and decommission approval — breaks. The fix is not remediation after the fact. The fix is a policy layer that prevents untagged resources from being created at all. AWS provides two complementary controls for this: SCPs at the organization level, which deny the API call before it reaches the service, and AWS Config rules, which flag non-compliant resources within minutes of creation.
SCP condition key mechanics
A Service Control Policy is a JSON policy document attached to an AWS Organizations organizational unit that acts as a permission ceiling. No IAM role in the affected account, regardless of its attached policies, can exceed what the SCP allows. Attaching a Deny condition on ec2:RunInstances when the request lacks a aws:RequestTag/Owner tag means the API call never reaches EC2. The resource is never provisioned.
There is no orphan to clean up later.
SCP tag enforcement. The SCP Deny block must reference aws:RequestTag, not aws:ResourceTag. The RequestTag condition key evaluates the tag supplied at creation time. The ResourceTag condition key evaluates tags already on the resource, which means it does nothing at creation. Getting this wrong produces an SCP that appears to enforce tagging but never fires.
We built this mistake in our first governance rollout and discovered it only after 30 days of data showed zero SCP denials against a team that was visibly creating untagged resources.
Tag schema and IaC gates
AWS Config as the second gate. SCPs cover services that support aws:RequestTag in their IAM condition keys. Not every AWS service does. AWS Config's managed rule required-tags evaluates all resources of a specified type and marks any instance missing the required key as NON_COMPLIANT. Config delivers this finding within minutes of creation because it evaluates on configuration change events, not on a polling schedule.
The finding routes to Security Hub, which routes to the owning team's alert channel. This works when the team has a defined alert destination. It breaks when the account has no Security Hub integration, because the finding sits in Config with no active consumer.
Tag schema design. The tag key name must be enforced by convention before the SCP references it. A team using owner and another using Owner produce two distinct tag keys. The SCP condition is case-sensitive, so owner: alice passes a check for Owner and the resource is denied even though intent was correct. Define one canonical key in a written standard before deploying the SCP.
Rollout and validation approach
In our environment, we standardized on Owner with a value format of team-name/iam-username, which made both cost allocation and incident routing unambiguous.
IaC pipeline integration. The most reliable enforcement point is the deployment pipeline, not the AWS API boundary. When Terraform or CloudFormation templates are validated in CI before plan or deploy runs, a missing Owner tag fails the build at $0 cost, before any AWS API call is made. The SCP is the backstop for console-created resources and manual API calls. An m5.xlarge instance running on-demand costs USD 185 per month.
A single idle instance created through the console by an engineer bypassing IaC represents USD 185 per month of unattributable spend. At 13 such instances, that is USD 2,405 per month that no team will claim ownership of during a cost review.
| Control | Enforcement Point | Failure Condition |
|---|---|---|
SCP with aws:RequestTag
|
AWS Organizations, before API reaches service | Service does not support RequestTag condition key |
AWS Config required-tags
|
Post-creation, within minutes | No Security Hub integration to consume findings |
| CI pipeline tag validation | Before terraform plan or deploy runs |
Engineers bypass IaC and use the console directly |
| Tag schema standard | Written convention before SCP deployment | Case mismatch between teams on key name |
Start with the SCP in a single non-production OU for the first two weeks. Measure the denial rate daily. A denial rate above zero in week one confirms engineers are creating resources without tags through console or direct API calls, and that population is exactly what the SCP is designed to eliminate before it reaches production accounts.
Building a Sustainable Ownership Accountability Practice
Orphan accountability fails permanently without a repeating audit cycle, because AWS environments accumulate untagged and unclaimed resources continuously as teams rotate and projects end.
Designing the audit cadence
A one-time cleanup produces a clean state that degrades within weeks. The mechanism is straightforward: engineers leave, pipelines change, and the SCP enforcement you built last quarter does not retroactively tag the 40 resources created before it was deployed. Governance requires a scheduled loop, not a project.
We built a 30-day audit cadence as the baseline. Each cycle runs three checks in sequence: CloudTrail attribution for resources created in the prior period, Config compliance status for tag coverage, and a cost allocation report filtered to resources with no Owner tag value. Resources that fail all three checks enter a formal orphan queue.
Automated alerting cadence. Each resource entering the orphan queue generates a ticket assigned to the last-known IAM principal from CloudTrail. The ticket carries a 7-day response window. This works when the IAM principal is still an active employee. It breaks when the principal belongs to a departed engineer, because the ticket routes to a dead inbox and sits unresolved.
Scoring and terminating orphans
The fix is a secondary assignment rule: if the IAM principal's account is disabled in your identity provider, the ticket routes to the engineering manager of the team that owned the principal's last active project.
Team accountability scoring. Tracking orphan counts per team over time creates the accountability signal that quarterly reviews need. We measured orphan rate per team as a ratio of untagged resources to total resources owned. A team with a ratio above 0.10 after 30 days of remediation time receives a formal review. Below 0.05 is the target.
This works because it converts an abstract governance problem into a number that appears in engineering manager dashboards, where it competes for attention alongside delivery metrics.
Termination authority. Resources that remain unowned after two full audit cycles, meaning 60 days with no claimed owner, enter a scheduled termination workflow. The workflow snapshots the resource if applicable, then terminates it. An m5.xlarge running unowned for 60 days costs USD 370 at on-demand pricing before termination. Multiply by the number of unclaimed instances in a typical enterprise account and the financial case for enforcement writes itself.
Preventing destructive false positives
This works when termination authority is pre-approved by leadership. It breaks when every termination requires a manual approval gate, because the queue backs up and the 60-day window becomes meaningless.
The Blast Radius Score. Before terminating any orphan, assign it a Blast Radius Score: a simple three-factor check covering active network connections, attached storage with recent write activity, and membership in a load balancer target group. A resource scoring positive on any factor gets a 14-day hold and a direct Slack alert to the account's on-call rotation rather than immediate termination. Zero-score resources terminate on schedule. This named check prevents the single most common governance failure mode: destroying a resource that turned out to be load-bearing despite having no owner tag.
| Governance Checkpoint | Frequency | Failure Condition |
|---|---|---|
| CloudTrail attribution sweep | Every 30 days | Trail was inactive during resource creation period |
| Config tag compliance report | Continuous, reviewed monthly | No Security Hub consumer routing findings to teams |
| Orphan queue ticket assignment | Per resource, on queue entry | IAM principal account is disabled in identity provider |
| Blast Radius Score check | Before every termination | Network and storage activity data is unavailable |
Run the first audit cycle manually before automating it. The manual pass exposes the gaps in your CloudTrail coverage and tag schema before the automation encodes those gaps into a repeating process.
Frequently Asked Questions
Q: How does the orphan problem: why aws resource ownership disappears apply in practice?
See the section above titled "The Orphan Problem: Why AWS Resource Ownership Disappears" for the full breakdown with examples.
Q: How does cloudtrail captures resource creation events apply in practice?
See the section above titled "How CloudTrail Captures Resource Creation Events" for the full breakdown with examples.
Q: How does querying cloudtrail to identify a resource's creator apply in practice?
See the section above titled "Querying CloudTrail to Identify a Resource's Creator" for the full breakdown with examples.
Q: How does enforcing ownership at creation time with tagging and scps apply in practice?
See the section above titled "Enforcing Ownership at Creation Time with Tagging and SCPs" for the full breakdown with examples.
Drop a comment if you've audited a similar spike. What was the dominant cause for your team? Share what worked or what blew up.






Top comments (0)