Originally published on Medium.
How to detect every EC2 launch and termination across an entire AWS Organization, and turn it into automated DNS registration, without deploying repeated infrastructure in every account.
I spent longer than I would like to admit staring at this problem before the answer got simple. So if you are wiring up DNS across an AWS Organization right now, maybe this saves you a weekend.
Here is the short version of what I landed on. I needed near real-time DNS registration for every EC2 instance across a multi-account, multi-region org, feeding an on-prem enterprise DDI system (think Infoblox). The two obvious ways to do it both annoyed me: one puts infrastructure in every account, the other leans on a fragile cross-account API-scraping job.
What I used instead is a hub-and-spoke event setup:
- Each spoke account gets exactly one EventBridge rule. That is the whole footprint.
- One hub account does everything else: a custom event bus, an SQS queue, one Lambda.
- The Lambda fills in the missing details by querying the AWS Config Aggregator, which, and this is the part I almost missed, already has every instance's IP, tags, and state for the entire org. No new data pipeline required.
Onboarding a new account came down to a one-line deploy. Below is the whole thing: the architecture, why I made the calls I made, and the code.
The problem (why the single-account version does not scale)
In one account this is a five-minute job. EC2 fires a state-change event, you point an EventBridge rule at a Lambda, done. I have written that Lambda a dozen times.
Across an organization it falls apart for a few reasons:
- Instances come up in dozens of accounts and several regions.
- The people who run DNS are not the people who own the workload accounts, so "just add it to the account" is not really on the table.
- The EC2 state-change event does not include the IP address. You get an instance ID and a state. That is it. Useless for DNS on its own.
- Nobody wants to deploy, version, and babysit a Lambda plus queue plus IAM role in every account just to catch these events.
The usual reason you end up here is a move to an enterprise DDI platform. Every instance that boots needs a DNS record, every instance that dies needs its record cleaned up, org-wide, in near real-time, with as little junk in each account as you can manage.
So two things really drove the design:
- Adding an account should be almost free.
- The IP is not in the event, so I have to look it up, and that lookup cannot be flaky.
The approaches I threw out
Option A: put the whole stack in every account. Rule, queue, Lambda, IAM role, shipped everywhere with StackSets. It works. But now you own N copies of the same code, and every bug fix or IAM tweak is an N-account rollout. I have maintained fleets like that. No thanks.
Option B: a central poller that assumes a role into the source account and calls DescribeInstances for the IP. Logic is centralized, which is nice, but you need an assume-role trust in every account and you are paying a cross-account API hop on every single event. More IAM surface, more things to break.
Option C: hub-and-spoke, enriched from the Config Aggregator. This is the one. All the logic sits in the hub. Spokes just forward. And instead of building anything to fetch the IP, the hub asks the Config Aggregator, which is already sitting there aggregating EC2 config (IP, tags, state) from every account and region.
Option C won for a boring but decisive reason: the hard part, the enrichment data, already existed. I did not have to build it. I just had to notice it was there.
How it fits together
Spokes forward EC2 state changes to the hub's custom bus; the hub enriches via the org Config Aggregator, then calls the DDI system. Spokes are dumb, the hub is smart, and the data already exists. That is the whole idea in one line.
That second hub rule is easy to forget, by the way. If you only listen on the custom bus, the hub account's own instances never get DNS records, because they never leave home to reach the custom bus. Ask me how I know.
Filling in the blanks
Most single-account tutorials skip this bit, which is exactly the bit that matters at scale. Here is what the event actually gives you:
{
"source": "aws.ec2",
"detail-type": "EC2 Instance State-change Notification",
"detail": {
"instance-id": "i-1234567890",
"state": "running"
}
}
An ID and a state. No IP, no tags, no hostname. So the Lambda goes and gets the rest, from the aggregator, not from the source account.
The nice thing: the aggregator is org-level, so one query resolves an instance no matter which account or region it lives in. No assume-role, no per-account permissions to manage.
The code
Everything here is vendor-neutral and anonymized. Swap the Infoblox call for whatever you actually talk to: Route 53, a CMDB, an IPAM API.
The spoke's one and only rule (Terraform)
This is the entire per-account footprint. Ship it with StackSets or a Terraform module.
# --- Deployed in EVERY spoke account/region ---
resource "aws_cloudwatch_event_rule" "forward_ec2_state" {
name = "forward-ec2-state-to-hub"
description = "Forward EC2 running/terminated events to the hub account"
event_bus_name = "default"
event_pattern = jsonencode({
source = ["aws.ec2"]
"detail-type" = ["EC2 Instance State-change Notification"]
detail = {
state = ["running", "terminated"]
}
})
}
resource "aws_cloudwatch_event_target" "to_hub_bus" {
rule = aws_cloudwatch_event_rule.forward_ec2_state.name
event_bus_name = "default"
arn = "arn:aws:events:${var.region}:${var.hub_account_id}:event-bus/ec2-dns-hub-bus"
role_arn = aws_iam_role.eventbridge_cross_account.arn
}
# Minimal role that lets EventBridge PutEvents to the hub bus
resource "aws_iam_role" "eventbridge_cross_account" {
name = "eventbridge-forward-to-hub"
assume_role_policy = jsonencode({
Version = "2012-10-17"
Statement = [{
Effect = "Allow"
Principal = { Service = "events.amazonaws.com" }
Action = "sts:AssumeRole"
}]
})
}
resource "aws_iam_role_policy" "put_events" {
role = aws_iam_role.eventbridge_cross_account.id
policy = jsonencode({
Version = "2012-10-17"
Statement = [{
Effect = "Allow"
Action = "events:PutEvents"
Resource = "arn:aws:events:${var.region}:${var.hub_account_id}:event-bus/ec2-dns-hub-bus"
}]
})
}
The hub's custom bus, with an org-scoped resource policy
This policy is the thing that makes "one rule per spoke" actually work: any account in the org can publish, and the hub needs zero per-account config.
# --- Deployed ONCE in the hub account ---
resource "aws_cloudwatch_event_bus" "hub" {
name = "ec2-dns-hub-bus"
}
resource "aws_cloudwatch_event_bus_policy" "allow_org" {
event_bus_name = aws_cloudwatch_event_bus.hub.name
policy = jsonencode({
Version = "2012-10-17"
Statement = [{
Sid = "AllowOrgAccountsToPutEvents"
Effect = "Allow"
Principal = "*"
Action = "events:PutEvents"
Resource = aws_cloudwatch_event_bus.hub.arn
Condition = {
StringEquals = {
"aws:PrincipalOrgID" = var.aws_org_id
}
}
}]
})
}
Do not skip the aws:PrincipalOrgID condition. It scopes that Principal = "*" down to your organization instead of, you know, the entire internet. It is the one line standing between "elegant pattern" and "open event bus."
Rules, queue, and Lambda wiring (hub)
resource "aws_sqs_queue" "events" {
name = "ec2-dns-events"
visibility_timeout_seconds = 120
redrive_policy = jsonencode({
deadLetterTargetArn = aws_sqs_queue.dlq.arn
maxReceiveCount = 5
})
}
resource "aws_sqs_queue" "dlq" {
name = "ec2-dns-events-dlq"
}
# Rule on the custom bus (events from spokes)
resource "aws_cloudwatch_event_rule" "hub_custom" {
name = "ec2-state-custom-bus"
event_bus_name = aws_cloudwatch_event_bus.hub.name
event_pattern = jsonencode({
source = ["aws.ec2"]
"detail-type" = ["EC2 Instance State-change Notification"]
detail = { state = ["running", "terminated"] }
})
}
resource "aws_cloudwatch_event_target" "custom_to_sqs" {
rule = aws_cloudwatch_event_rule.hub_custom.name
event_bus_name = aws_cloudwatch_event_bus.hub.name
arn = aws_sqs_queue.events.arn
}
# Second rule on the default bus catches the hub account's OWN instances
resource "aws_cloudwatch_event_rule" "hub_default" {
name = "ec2-state-default-bus"
event_bus_name = "default"
event_pattern = jsonencode({
source = ["aws.ec2"]
"detail-type" = ["EC2 Instance State-change Notification"]
detail = { state = ["running", "terminated"] }
})
}
resource "aws_cloudwatch_event_target" "default_to_sqs" {
rule = aws_cloudwatch_event_rule.hub_default.name
arn = aws_sqs_queue.events.arn
}
resource "aws_lambda_event_source_mapping" "sqs_to_lambda" {
event_source_arn = aws_sqs_queue.events.arn
function_name = aws_lambda_function.dns_registrar.arn
batch_size = 10
}
The enrichment Lambda (Python)
This is the heart of it: resolve the instance against the Config Aggregator, then act on the state.
import json
import boto3
config = boto3.client("config")
AGGREGATOR_NAME = "OrganizationConfigAggregator"
def get_instance_details(instance_id: str) -> dict | None:
"""Resolve IP, tags, account, and region from the org Config Aggregator."""
expression = (
"SELECT resourceId, accountId, awsRegion, "
"configuration.privateIpAddress, "
"configuration.publicIpAddress, tags "
f"WHERE resourceId = '{instance_id}' "
"AND resourceType = 'AWS::EC2::Instance'"
)
resp = config.select_aggregate_resource_config(
Expression=expression,
ConfigurationAggregatorName=AGGREGATOR_NAME,
Limit=1,
)
results = resp.get("Results", [])
return json.loads(results[0]) if results else None
def handler(event, _context):
for record in event["Records"]:
body = json.loads(record["body"])
detail = body["detail"]
instance_id = detail["instance-id"]
state = detail["state"]
details = get_instance_details(instance_id)
if not details:
# Terminated instances may already be gone from the aggregator;
# fall back to event data for deregistration.
if state == "terminated":
deregister_dns(instance_id)
continue
ip = (
details.get("configuration", {}).get("privateIpAddress")
or details.get("configuration", {}).get("publicIpAddress")
)
if state == "running" and ip:
register_dns(instance_id, ip, details.get("tags", []))
elif state == "terminated":
deregister_dns(instance_id)
def register_dns(instance_id, ip, tags):
# Replace with your Infoblox / Route 53 / IPAM call.
# Infoblox example: POST to the WAPI record:a endpoint.
print(f"REGISTER {instance_id} -> {ip}")
def deregister_dns(instance_id):
print(f"DEREGISTER {instance_id}")
One thing worth calling out: that SELECT against the aggregator resolves a resource across every account and region in a single call. That is the capability that collapses the whole cross-account enrichment headache into a one-liner. I keep coming back to it for other problems now.
And notice the fallback in the handler. A terminated instance has often already dropped out of the aggregator by the time you look, so you cannot rely on the lookup for deregistration. You fall back to the instance ID from the event. Small thing, but it will bite you in testing if you miss it.
Why there is an SQS queue in the middle
You can target the Lambda straight from the rule, and for a demo I would. In production I put SQS in between on purpose:
- A bulk launch, a scale-out, a StackSet rollout, can throw hundreds of events at you at once. SQS soaks that up and lets the Lambda drain at its own pace instead of getting hammered.
- When Infoblox or the aggregator has a bad minute, messages retry and eventually land in a DLQ you can actually look at. Nothing quietly disappears.
- Batching. Ten events per invocation, fewer cold starts, lower cost.
Decoupling ingest from processing is most of what separates "worked in the demo" from "still working at 2am during a mass launch."
What actually came out of it
- Detection and DNS updates happened within seconds, across the whole org.
- Adding an account or region was a single forwarding rule. No Lambda, no queue, no Config setup per account.
- Registration and cleanup were fully automated across the instance lifecycle, nobody touching DNS by hand.
- Cost tracked instance activity, not the clock, because there is no idle compute.
- The shape is reusable. Same hub-spoke plus aggregator-enrichment works for CMDB sync, security tagging, IPAM updates, anything org-wide and event-driven.
The part worth stealing
Take Infoblox out of the picture and you are left with a pattern I now reach for often:
Fan org-wide events into one hub, then enrich them from a data source the organization already has, instead of building a new one.
Two moves do the work:
- A custom event bus with an
aws:PrincipalOrgIDresource policy gives you cross-account fan-in and a one-rule spoke footprint. - The Config Aggregator, used as an enrichment lookup, resolves any resource in any account or region without an assume-role fan-out.
So the next time you are about to copy the same Lambda into 40 accounts, stop and ask whether a fan-in plus an aggregator you already run could do it with one rule per account instead. Usually it can.
Wrapping up
Honestly, none of the individual pieces here are clever. Cross-account EventBridge is documented. Config aggregators are documented. The win was in composing them so the per-account cost dropped to near zero and the enrichment data was something we already had lying around. That is where most of the good architecture calls seem to live, not in the shiny new component, but in the one you talked yourself out of building.
If you end up adapting the enrichment step to Route 53, a CMDB, or your own IPAM, I am happy to talk through it in the comments.
Top comments (5)
The use of the Config Aggregator for cross-account enrichment is the real hero here. Most people forget that the aggregator already has all that instance data and spend weeks building a custom cross-account scraping pipeline instead.
Two technical questions:
You mentioned registering DNS from the running state. Do you handle the case where the instance gets a new IP after a stop/start cycle? Does the aggregator show the updated IP in time, or does the Lambda need to listen for a specific IP-change event?
For the hub account's own instances, are they also caught by the spoke rule (since the hub also has a default bus), or does the second rule create duplicate DNS registration attempts?
Great write-up on the "one rule per account" pattern!
Thank you, @alexcodebytes , and great question.
AWS stop/start documentation says private IPv4 addresses persist across a stop and start. The auto-assigned public IPv4 address is lost, and a new one is assigned on start unless an Elastic IP is attached. The instance goes through running again on every start, so the rule fires and the Lambda registers again. The handler prefers the private address, so for private DNS nothing changes. Only public-address registration is affected.
The risk is timing, not a missing event. AWS Config records changes "right after a change is detected", but on a best effort basis that can take longer. The EC2 state-change events are also best effort. So the aggregator can briefly show an older state or IP, or not show a new instance yet. The handler in the article skips a running instance when the lookup returns nothing, and I should have covered that.
I think:
Raise an error instead of skipping when the lookup is empty or older than the event. Compare the result's configurationItemCaptureTime (a selectable property) with the event time.
Let SQS retry. Lambda returns the whole batch to the queue on a failure, so turn on partial batch responses to retry only the failed messages.
Run a scheduled reconciliation that compares running instances in the aggregator with DNS, since events can be missed.
Hub account. A rule applies only to events received by its own bus. The spoke rule forwards to the hub's custom bus, and the hub's own EC2 events stay on its default bus, so the two rules do not overlap. Duplicates can happen if the spoke rule is also deployed in the hub account, so exclude the hub from the StackSet. Standard SQS queues can deliver a message more than once, so make DNS registration idempotent, keyed on the instance ID.
The 'raise an error instead of skipping' point is the most important one here. A silent skip is exactly how you end up with a stale DNS record that nobody notices until something breaks. Comparing configurationItemCaptureTime with the event time is a clean way to make that check explicit.
@alexcodebytes. Agreed.
A silent skip is the worst outcome because nothing fails and the record just goes stale.
One note on the capture-time check: it is a heuristic, not a guarantee. The capture time says when Config recorded the item, so it only shows that the lookup is not older than the event. It cannot tell you that Config has caught up on every other change.
The retry window also has a limit. With the queue settings in the article (120 second visibility timeout, five receives), a message gets roughly eight minutes of attempts before it lands in the dead-letter queue. That is why I would alert on the DLQ and keep the scheduled reconciliation as the final safety net.
I really appreciate you calling out the retry window math. 'Roughly eight minutes of attempts before the DLQ' is a concrete number that makes the trade-off visible. It's the kind of detail that turns a tutorial into a real operational guide.