A CloudWatch CPU alarm driving Systems Manager remediation through EventBridge, in two CloudFormation templates: a runbook and a direct Run Command target.
An alarm on CPUUtilization above 90 percent, an EC2 instance registered as a managed node, and a shell script that has to run on the instance that breached. That is the whole scope, and the design everyone reaches for puts a CloudWatch alarm, an Amazon EventBridge rule, and an AWS Systems Manager Automation runbook between the first two and the third.
Run by hand, the same procedure is three steps and a judgement call: read the alarm, find the instance behind the data point, send the script to that instance and nothing else. The judgement call is the step that does not survive being automated carelessly, because it is the one that decides where the script lands. Send it to the wrong node and the remediation becomes the incident.
The common design works. What it hides is a single assumption: that the identifier of the instance that breached travels from the metric to the action. Three of the six designs on the table cannot carry it at all, and the three that can carry it in three different ways, at three different prices. One of them pays with a host on the critical path, one pays with code to own, and the one I kept carries the value only because a specific JSON path happens to exist inside the alarm's own configuration snapshot. That path is the whole decision, and it is also the structural limit of the design.
The condition, and the procedure that has to run
The condition is a critical application running on EC2 instances where, during certain periods, CPU utilization climbs above 90 percent and application performance degrades. The operations team already watches CPUUtilization in CloudWatch. Nobody is arguing about the metric.
The procedure is also settled, and that matters more than it looks. There is a predefined script, it runs on the affected instance, and it runs through Systems Manager Run Command. The instances are already registered as managed nodes. The script is not up for redesign, which means the design space is narrower than a blank page: whatever gets built has to end in a Run Command invocation on a specific node.
The stated constraints are AWS managed services and the least operational overhead. Two constraints that sound like the same thing and are not. Managed services rules out owning a host. Least operational overhead rules out owning a workflow nobody reads.
A few numbers that come from the design rather than from any measurement, because they reappear later. EC2 publishes CPUUtilization in 5-minute data points by default and in 1-minute data points with detailed monitoring enabled. The alarm in this post evaluates 3 breaching data points out of 3 periods of 60 seconds, which is a choice, not a default. Multiply those two and the floor between the load starting and the alarm transitioning is three minutes, plus the metric publishing delay. That number is arithmetic from the configuration; the lab measured 245 seconds against it, and the section on alarm delay shows where the extra 65 seconds came from.
Requirements for the remediation path
- Detection reads
CPUUtilizationin theAWS/EC2namespace with theInstanceIddimension, and needs no extra agent on the node. - Remediation runs on the instance that presented the condition, identified at run time from the event, not from a static list.
- No new compute resource participates in detection or in the decision.
- The corrective script terminates only the identified load, by unit name, and is idempotent: running it twice produces the same result as running it once.
- An invocation failure leaves recoverable evidence, with an error code and a payload, instead of disappearing.
- A remediation execution that fails produces a notification rather than silence.
- The role EventBridge assumes can start that runbook and nothing else, and the Automation execution role can send that one command document, and only to nodes carrying the tag value this stack itself produces.
- No long-lived credential, and no inbound port on the instance. The SSM Agent opens every connection.
- Everything is born from CloudFormation, and teardown is one line.
- The whole path is observable through metrics and alarms: invocation failure, dead letter queue depth, failure to write to the dead letter queue, and script failure.
Six designs for the same remediation
Six, and two of them are not straw men. Sending Run Command as a direct EventBridge target is a real design that the documentation supports, I ship it as the second template, and it is where the most useful finding in this post lives. Invoking a Lambda function as the alarm action is the design the CloudWatch documentation itself points at for custom actions on a state change, so it has to be argued against rather than left out.
Polling CloudWatch from a dedicated monitoring instance
The argument in favor is stronger than it gets credit for. Full control, no cross-service integration to learn, no event shape to decode, no role to design. The script can do anything before and after the remediation: query a database inside the VPC, correlate with application latency, decide on more than one metric at a time. It works identically outside AWS, which has real value in a hybrid estate, and the team already knows how to write a script and edit a crontab.
What breaks is the host. A single instance sits on the critical path of the remediation. If it rebooted, degraded, is halfway through a patch, or was replaced, the polling loop does not run and nothing reports the absence. That is the decisive failure, a Reliability failure, and it arrives with the irony that defines the design: the monitoring instance is itself an EC2 instance subject to CPU saturation, and a monitor with saturated CPU stops monitoring at exactly the moment monitoring matters. The polling loop compounds it, because CloudWatch delivers alarm state changes to EventBridge and the documentation states that it guarantees that delivery, so a loop calling GetMetricStatistics every N seconds trades a guaranteed push for a sampling interval, a Performance Efficiency failure that lands inside the reliability one because the sampling interval is only as available as the host doing the sampling.
The second failure is Operational Excellence. Someone patches the host, someone owns a script nobody reviews, and the execution history becomes whatever the script chose to log. Compare that with what the managed path emits for free, across three event types with distinct payloads: EC2 Automation Execution Status-change Notification carries ExecutionId, Definition, Status, StartTime and EndTime, EC2 Automation Step Status-change Notification carries StepName and Action, and EC2 Command Invocation Status-change Notification carries instance-id, command-id and status. On top of that, CommandId, Status, ResponseCode and Output come back as outputs of the action itself.
The third failure is Security. An SSH channel needs a long-lived credential or a key to distribute and rotate, and it needs an inbound port on the remediated instances. The Systems Manager documentation records the opposite arrangement explicitly: the SSM Agent initiates all connections to the service, and for that reason there is no need to configure the firewall to allow inbound traffic to the instances. Trading an IAM-authenticated outbound path for a key-authenticated inbound path makes the posture worse in exchange for familiarity.
Cost and sustainability come after, as consequence rather than cause. The instance stays up 168 hours a week to react to events that last minutes, which is a Cost Optimization failure in the form of a resource billed by time for work measured in seconds, and a Sustainability failure in the form of allocated hardware sitting almost entirely idle. Neither is the reason to reject the design. They are what the design costs after it already failed on reliability.
An instance earns its place when the decision to remediate depends on state, on a heavy library, or on something that exists only inside the VPC. Not the case here: the decision is a threshold on a metric that EC2 already publishes.
A log metric filter calling Run Command
This one starts as a factual error rather than a pillar failure, and the order matters. A pillar does not apply to a capability that does not exist.
The argument in favor is real. A metric filter is the right tool when the signal exists only in a log line, and anyone who has built an alarm on an application error rate built it exactly this way: the filter extracts the value, publishes a metric, the alarm evaluates it. The vocabulary is correct and the reflex is good.
What a metric filter does is described in closed form. Its elements are a filter pattern, a metric name, a metric namespace, a metric value, a default value and dimensions. The result is a CloudWatch metric data point that you can graph or alarm on. There is no destination, no action, no invocation. The CloudFormation resource confirms it by absence: AWS::Logs::MetricFilter carries exactly ApplyOnTransformedLogs, EmitSystemFieldDimensions, FieldSelectionCriteria, FilterName, FilterPattern, LogGroupName and MetricTransformations, and no target, action or invocation property at all.
Where the capability would actually come from is worth naming, because it is two different things. To deliver log events to a service that executes code, the resource is a subscription filter, not a metric filter. To turn a numeric value found in a log into an action, the path is metric filter, then metric, then alarm, then the same EventBridge hop this post builds. The metric filter is never the last link.
The second factual error runs deeper. CPUUtilization does not come from a log. The EC2 documentation defines it as the percentage of allocated physical CPU time that the instance uses, covering user code and EC2 code, and describes the value at a high level as the sum of guest and hypervisor utilization. EC2 publishes it to the AWS/EC2 namespace. No log line carries that number, so no filter pattern can extract it. To get CPU as the operating system sees it into CloudWatch you install the CloudWatch agent, and even then the result is a metric, not a log.
That same page records something worth one sentence of its own: operating system tools can report a different percentage from CloudWatch, for reasons including legacy device simulation, interrupt-heavy workloads, live migration and live update. Anyone comparing top against the CloudWatch graph and concluding that one of them is wrong is reading two measurements of different things.
Only now does the corrected design earn a pillar. Log, then metric filter, then metric, then alarm, then remediation is longer than metric, then alarm, then remediation, and it inserts a log ingestion dependency between the condition and the response. That is an Operational Excellence failure: one more place where the path can stop without anything changing state, in a path whose entire purpose is to react. Add to it that metric filters are supported only on Standard class log groups, that filters are not retroactive, and that delivery of log events to the metric is at least once, with occasional duplicates.
The runbook template in this post does use a metric filter, and it uses it the right way: counting failures of the remediation script from the command output in CloudWatch Logs. There the signal exists only as a line of text, and the metric filter is exactly the tool. The difference between that use and the one proposed here is the lesson. The second template has no metric filter at all, for a reason that only shows up when the command output has nowhere to land, and that comes later.
An Auto Scaling policy restarting the instance
This one also starts as a factual error.
The argument in favor is the most defensible of the six. Ninety percent CPU is often a capacity problem, and capacity is exactly what Auto Scaling solves. Replacing an instance is a legitimate and widely used recovery strategy, and running configuration at launch is an established pattern. Anyone proposing this is thinking in terms of immutable infrastructure.
What a scaling policy does: a dynamic scaling policy tracks a CloudWatch metric and defines which action to take while the associated alarm is in ALARM, and step and simple scaling policies scale group capacity in predefined increments. The action is capacity. There is no restart action in a scaling policy.
What Auto Scaling does when it considers an instance unhealthy is replace it, and replacement is terminate and launch, not restart: by default it creates a scaling activity to terminate the unhealthy instance and terminates it, and while that instance is terminating another activity launches a new one. The trigger for that is a health check, not a scaling policy.
Restart and recover actions live somewhere else entirely. They are CloudWatch alarm actions on EC2 metrics, not Auto Scaling actions. And automatic instance recovery addresses a system status check failure, is monitored through StatusCheckFailed_System, and the documentation is explicit that it does not operate when only the instance status check fails. High CPU triggers none of that.
The third problem is what makes the design inapplicable even after the corrections. The metrics that drive a scaling policy's alarm into ALARM are an aggregation of the metrics from all instances in the Auto Scaling group. One hot instance in a group of ten may never move the group average above 90 percent. If it does, the group scales out, which addresses capacity and leaves the stuck process exactly where it was.
Now the pillar. Replacing an instance is not equivalent to executing a specific corrective procedure, and running the script at launch runs it on the new instance, which by definition does not have the condition. That is an Operational Excellence failure: the corrective procedure stops being a named action with a CommandId, a Status, a ResponseCode and an execution history, and becomes a side effect of a capacity change. It is also a Reliability failure because of in-flight work: the Auto Scaling documentation records that by default it has no visibility into the applications running on the instances and may terminate them before the application can shut down gracefully or finish the work assigned to it. Trading a script that takes seconds for a termination picks the most destructive remedy available.
Scaling is the right answer when the 90 percent is legitimate demand. That bridge gets built later in this post.
An alarm driving Run Command as a direct EventBridge target
The argument in favor: this is the shortest possible path. EventBridge accepts Systems Manager Run Command as a target and calls SendCommand on your behalf. One hop fewer than the runbook, one role fewer, one document fewer, no Automation step to version, and Run Command is offered at no additional charge. For a single-step procedure, insisting on an Automation runbook looks like exactly the kind of layer worth removing.
What fails is documented rather than a matter of taste. The Run Command target is configured through RunCommandParameters, whose RunCommandTargets accepts a Key of InstanceIds with a list of IDs, or tag:<key> with tag values. The document and its parameters go in the target's Input field, which is valid JSON passed to the target and in which nothing from the event is passed. And dynamic path parameters, the syntax that would let you pull a value out of the event at run time, are supported in the non-enum string fields of EcsParameters, HttpParameters, RedshiftDataParameters, SageMakerPipelineParameters and SqsParameters. RunCommandParameters is not on that list.
The consequence, stated plainly: with Run Command as a direct target, the target set is static or tag-based. If it is tag-based, every alarm transition sends the script to the whole tagged set, not to the node that breached. In a group of ten instances, one hot instance produces ten executions, nine of them on a healthy node. That is an Operational Excellence failure, because the corrective action loses the property of being targeted and the history now records nine executions that should not have happened, and it becomes a Reliability failure the moment the corrective script is not harmless on a healthy node, because the blast radius of the remediation becomes the entire set.
And the tagged set is wider than it looks, which turns the same Reliability failure up a notch. A tag target is resolved against the managed nodes of the account and the Region at invocation time. There is no stack boundary, no VPC boundary and no subnet boundary in that resolution, because Run Command reaches the node through the Systems Manager API and not through the network the node sits on. So the blast radius of a tag-targeted remediation is every node in the account and Region wearing that tag, including nodes created by a different team, a different template or a copy of this one, and the way to keep it narrow is to make the tag value unique per deployment rather than to rely on the lab being small.
The honest mitigation exists, which is why the second template ships instead of being discarded. If the script is strictly idempotent and harmless when the condition is absent, remediating by tag is acceptable and simpler. The script in this post is written that way on purpose. What changes is what you are allowed to claim: with a direct target you remediate a set, not a node.
It is not perfect even with the mitigation. There is no aws:runCommand step, so there is no step, no onFailure, and no chaining. The command result lives in the EC2 Command Status-change Notification and EC2 Command Invocation Status-change Notification events, and that is all. And the target cannot even route the command output to CloudWatch Logs, which costs more than it sounds like and gets its own paragraph where the second template is built.
A Lambda function as the alarm action
The argument in favor comes from AWS. The CloudWatch documentation lists invoking a Lambda function among the supported alarm actions and calls it the easiest way to automate custom actions on alarm state changes. The function gets what it needs: the payload CloudWatch delivers carries an alarmData object with the alarm's configuration, so the breaching instance's dimension is in reach, and the function can call SendCommand with that instance ID. CloudWatch guarantees an asynchronous invocation for a given state change, except when the function does not exist and when CloudWatch is not authorized to invoke it, and it retries until the invocation succeeds if it cannot reach the Lambda service. That is one hop fewer than routing through EventBridge, and it is the design that can branch: inspect two metrics, decide between two procedures, give up and open an item instead.
What fails is the constraint, not the mechanism. The requirement is AWS managed services with the least operational overhead, and this design answers it with an artifact I own. A function is code to package, a runtime version to keep current, a log group of its own, a resource policy, an execution role, a concurrency setting, and a second place where remediation logic lives, next to the command document that already holds the script. That is an Operational Excellence failure: the shortest path acquires the longest lifecycle, and the thing being maintained duplicates what two managed services already express declaratively, an event pattern and a step. The failure is not hypothetical maintenance. It is that the next person debugging a remediation that did not fire now has three places to look instead of two, and one of them is source code.
There is a Security edge with a documented sharp side. A Lambda alarm action needs a resource policy on the function allowing the CloudWatch service principal to invoke it, and the documentation advises against a policy that does not specify a source account, because such policies are vulnerable to the confused deputy problem. The scoped version names the account, and the narrower version names the single alarm ARN. Nothing about that is hard, and it is one more thing that has to be right in a path whose whole point is to work unattended.
A function earns the alarm action when the decision needs logic an event pattern cannot express. For a single predefined script on a single node, it is a runtime standing where a parameter would do.
An alarm driving a Systems Manager Automation runbook
This is the design I would keep running, and it goes through the same analysis.
On Operational Excellence it wins because the runbook is an AWS::SSM::Document of type Automation, versioned in the same template as everything else, and each step emits its own event: EC2 Automation Step Status-change Notification carries StepName, Action, Status, StartTime and EndTime. The aws:runCommand action returns CommandId, Status, ResponseCode and Output as step outputs, so the script result becomes flow data rather than only a log line. The runbook also accepts onFailure and finallyStep, which makes error handling a declaration instead of a convention.
On Reliability it wins because no host sits in the path, RetryPolicy and DeadLetterConfig are configurable on the target itself, and the dead letter queue carries RULE_ARN, TARGET_ARN, ERROR_CODE, ERROR_MESSAGE, EXHAUSTED_RETRY_CONDITION and RETRY_ATTEMPTS as message attributes, which turns an invocation failure into a diagnosis instead of an absence.
On Cost Optimization it wins because nothing in the control path is billed for sitting idle between incidents: the alarm, the rule, the runbook and Run Command are all billed per evaluation, per event or per step, and an architecture that fires a handful of times a month is charged for a handful of firings. That is the comparison that matters against a monitoring host, which is billed for all 168 hours of a week to be ready for a few minutes of work. It is not a claim that the stack contains no time-billed resource: the customer managed KMS key costs USD 1 per month prorated hourly for as long as it exists, and the lab instance is billed by the hour. Those two sit in the lab, not in the control path, and the key is a deliberate choice discussed later.
And it is the only design where the identifier of the breaching instance reaches a targeted action without a host and without code of my own in the path, through the input transformer that reads $.detail.configuration.metrics[0].metricStat.metric.dimensions.InstanceId and hands that value to the runbook's InstanceId parameter. That fact, not a preference for a richer architecture, is what decides.
Where it is not perfect, with the same weight:
The instance ID is not a first-class field of the event. It sits inside the alarm's configuration, not inside the description of what happened. If the alarm watches an aggregated metric across several instances, the InstanceId dimension does not exist, the JSON path does not resolve, and the input transformer documentation is clear about the result: a variable whose JSON path does not exist in the event is not created and does not appear in the output. That leaves a required parameter missing, and the Systems Manager documentation records the effect: without a value for each required parameter, the system creates the rule, but the rule does not run.
An alarm invokes actions only when it changes state. Remediation lowers the CPU, the alarm returns to OK, the load comes back, the alarm fires again. That is a remediation loop masking a capacity problem.
There is delay, and it is arithmetic: Period multiplied by DatapointsToAlarm, plus the metric publishing delay.
Duplicate execution is possible, which is where the idempotency requirement on the script comes from.
Automation is billed per step, at USD 0.002 per step executed and USD 0.00003 per second of script executed. The previous free tier of 100,000 steps and 5,000 seconds of script duration per month stopped being available to new customers and ended on December 31, 2025 for existing ones, per the announcement of August 14, 2025. For a one-step remediation the absolute amount is irrelevant; as a fact that many people have not updated, it is not.
The execution role is a target. Whoever can send AWS-RunShellScript to a node has arbitrary execution on it, which is why both templates create their own parameterless command document and scope the permission to that document's ARN.
The breaching instance ID has to reach the action
The criterion that decided this has a name: the identifier of the instance that breached has to reach the action. Three of the six designs cannot carry it, for three different reasons. The Auto Scaling policy works on a group aggregate, so it never knows which instance is hot. The metric filter never produces an action at all. The direct Run Command target does not accept dynamic path syntax in its target parameters.
Three can carry it, and the tie breaks on what each one charges for the delivery. The monitoring instance reads the identifier from the metric itself and pays with a host on the critical path of the remediation. A Lambda alarm action reads it from the alarm payload and pays with code, a runtime and a resource policy to maintain, against a requirement that asks for the least operational overhead. The Automation runbook moves the value from the event into the parameter of a targeted action with a declaration, through a mechanism the AWS documentation walks through step by step, and the artifacts it adds are a document and a role rather than a host or a package.
Two templates ship, not one. The first uses the runbook, and it is the one I would leave running. The second uses Run Command as a direct target, and it exists so the difference can be measured rather than taken on faith. They are alternatives, not complements, and they deploy under different stack names.
The remediation path, from data point to exit code
The success path, in order:
- The test load runs on the instance under a named systemd unit.
- EC2 publishes
CPUUtilizationto theAWS/EC2namespace with theInstanceIddimension. With detailed monitoring enabled, each data point covers one minute. - The alarm evaluates 3 of 3 data points above 90 percent and transitions from OK to ALARM.
- CloudWatch sends an alarm state change event to EventBridge, a delivery the documentation states is guaranteed.
- The rule matches
source: aws.cloudwatch,detail-type: CloudWatch Alarm State Change,detail.alarmName,detail.state.value: ALARManddetail.previousState.value: OK. - The input transformer extracts
$.detail.configuration.metrics[0].metricStat.metric.dimensions.InstanceIdand$.detail.alarmNameand assembles the runbook's parameter map. - EventBridge assumes the invocation role and calls
ssm:StartAutomationExecution. - Automation assumes the execution role named in
AutomationAssumeRoleand runs theaws:runCommandstep, withDocumentNamepointing at the stack's command document andInstanceIdscarrying the value from the event. - The SSM Agent on the instance runs the command document.
- Command output goes to the log group declared in
CloudWatchOutputConfig. - The step returns
CommandId,Status,ResponseCodeandOutput. - CPU falls and the alarm returns to OK.
The failure path, in the same level of detail. A target invocation failure retries according to RetryPolicy and then goes to the dead letter queue; failures caused by missing permission, a target that no longer exists, an invalid address or a DNS failure go straight to the queue with no retry attempt. FailedInvocations in AWS/Events, with the RuleName dimension, feeds an alarm, and because that metric is sent only when it is non-zero, the alarm uses TreatMissingData: notBreaching. InvocationsFailedToBeSentToDlq feeds the alarm that catches the worse case, the failure that could not even become a record. Queue depth comes from ApproximateNumberOfMessagesVisible in AWS/SQS. An Automation execution ending in a terminal state other than success matches a second rule, on EC2 Automation Execution Status-change Notification from source: aws.ssm, filtered on detail.Definition and detail.Status, and notifies an SNS topic. And a script that runs successfully but fails to resolve the condition emits a text marker, which the metric filter counts and an alarm picks up.
flowchart LR
LOAD["Test load under a named systemd unit"] --> EC2["EC2 instance with SSM Agent"]
EC2 -->|"CPUUtilization in AWS/EC2, 1-minute periods"| CW["CloudWatch metric"]
CW --> ALARM["CPU alarm, 3 of 3 datapoints above the threshold"]
ALARM -->|"CloudWatch Alarm State Change, OK to ALARM"| RULE["EventBridge rule on the default event bus"]
RULE -->|"input transformer reads configuration.metrics.0.metricStat.metric.dimensions.InstanceId"| INVROLE["EventBridge invocation role"]
INVROLE -->|"ssm:StartAutomationExecution"| RUNBOOK["Automation runbook, schemaVersion 0.3"]
RUNBOOK -->|"AutomationAssumeRole"| EXECROLE["Automation execution role"]
EXECROLE -->|"aws:runCommand, ssm:SendCommand scoped by tag"| DOC["Command document, aws:runShellScript"]
DOC --> AGENT["SSM Agent on the breaching instance"]
AGENT --> LOGS["Command output in CloudWatch Logs"]
LOGS --> FILTER["Metric filter on the failure marker"]
FILTER --> SCRIPTALARM["Script failure alarm"]
RULE -->|"retries exhausted, or no permission with no retry"| DLQ[("SQS dead letter queue, customer managed KMS key")]
RULE -.->|"FailedInvocations"| ALARMS["CloudWatch alarms"]
RULE -.->|"InvocationsFailedToBeSentToDlq"| ALARMS
DLQ -.->|"ApproximateNumberOfMessagesVisible"| ALARMS
RUNBOOK -->|"EC2 Automation Execution Status-change, Failed or TimedOut"| FAILRULE["Failure notification rule"]
FAILRULE --> SNS["SNS alert topic, customer managed KMS key"]Where the instance ID lives in an alarm state change event
The alarm state change event has no instance field. What it has is a snapshot of the alarm's configuration, and the identifier comes out of that. The path is $.detail.configuration.metrics[0].metricStat.metric.dimensions.InstanceId, which is exactly what the AWS walkthrough puts in the transformer's input path. The authority for the shape is the CloudWatch page on alarm events and EventBridge, which publishes the example state change payloads: configuration carries metrics, each item carrying metricStat with a metric that has dimensions, name and namespace, plus period and stat, alongside state and previousState.
The implication is the sentence this whole post exists to deliver: the identifier does not describe what happened, it describes what the alarm watches. If the alarm watches one instance, the value is there. If the alarm watches an aggregated metric across several instances, or a metric math expression, the InstanceId dimension is not in the snapshot and therefore not in the event. The input transformer documentation says what happens then: a variable whose JSON path does not exist in the event is not created and does not appear in the output. The result is a parameter map with no InstanceId. The Systems Manager documentation says what that produces: without a value for each required parameter, the system creates the rule, but the rule does not run.
So the architectural consequence, stated without hedging: this design is per instance. One alarm per node, and the number of alarms grows with the fleet. Anyone who wants a single alarm for the whole fleet cannot use this path to remediate the node that breached, because the node that breached is not in the event. That limitation does not appear in any condensed version of the design.
In the lab run, the value made the trip. The runbook execution that the transition started carries the breaching instance in its parameters, and the aws:runCommand step carries the same value in its inputs, which is the pair that proves the transformer did the extraction rather than the target guessing:
$ aws ssm get-automation-execution --automation-execution-id 70b6fea9-cec2-41c8-aaac-60cabdfe3921
Status: Success
Start: 2026-10-13T12:06:07.105000-03:00 End: 2026-10-13T12:06:17.804000-03:00
Parameters: {"AlarmName": ["postlab-ec2-remediation-runbook-workload-cpu-high"],
"InstanceId": ["i-0e60350eb60ad5cc0"]}
step: runRemediation action=aws:runCommand status=Success
Inputs: {"InstanceIds": "[\"i-0e60350eb60ad5cc0\"]",
"CloudWatchOutputConfig": "{\"cloudWatchLogGroupName\":\"/aws/ssm/postlab-ec2-remediation-runbook-remediation\",\"cloudWatchOutputEnabled\":true}",
"TimeoutSeconds": "120"}
Outputs: {"CommandId": ["868279b1-2cc6-4d63-99d5-bc2277b60e50"],
"Output": ["REMEDIATION_START unit=cpu-load-test.service instance=i-0e60350eb60ad5cc0 at=2026-10-13T15:06:08Z\nREMEDIATION_ACTION stopped=cpu-load-test.service\nREMEDIATION_OK cpu=0\n"]}
The stack's instance was i-0e60350eb60ad5cc0. That identifier appears in the runbook parameter, in the step input and inside the script output, and it is the only instance the command reached. The step: and action= labels in that block are mine, from the capture; the identifiers, timestamps and status values are the service's.
One syntax limit is worth half a sentence. EventBridge does not support the full JSON Path syntax and evaluates it at run time; what is supported is dot notation, hyphens, underscores, alphanumerics, array indices, wildcards and slashes. There is no filter, no conditional expression, no function. The path used here fits inside that subset by luck of design rather than by guarantee.
A CloudWatch alarm does not fire the moment CPU crosses 90 percent
Period is the interval over which the statistic is applied, with valid values of 10, 20, 30, 60 and any multiple of 60. But EC2 publishes CPUUtilization in 5-minute data points by default and 1-minute data points with detailed monitoring, and the minimum granularity EC2 provides for Minimum, Maximum and Average is one minute. A period shorter than the publishing interval accelerates nothing; it produces an alarm that spends most of its time without data.
EvaluationPeriods is the N and DatapointsToAlarm is the M of an M out of N alarm. Omitting DatapointsToAlarm makes CloudWatch use the same value as EvaluationPeriods, and the alarm goes to ALARM when that number of consecutive periods is breaching. TreatMissingData accepts breaching, notBreaching, ignore and missing, with missing as the default.
A newly created alarm goes immediately to INSUFFICIENT_DATA, is then evaluated, and only then takes the corresponding state, at which point any action associated with the new state is executed. An alarm invokes actions only when it changes state, with the exception of Auto Scaling actions, which continue to be invoked once a minute while the alarm stays in the new state. Alarm history is retained for 30 days, each transition with its own timestamp, and in rare cases the history can show more than one notification for the same change, in which case the timestamp is what lets you confirm unique transitions. Alarms also carry an evaluation range quota: the period multiplied by the number of periods used, capped at seven days for alarms with a period of at least one hour and one day for shorter periods.
With 3 of 3 data points of 60 seconds, the floor between the load starting and the alarm transitioning is three minutes, plus the metric publishing delay. The lab measured 245 seconds, and the minute-by-minute metric shows exactly where the difference lives. Load started at 2026-10-13T15:02:01Z, and the timestamps below are America/Sao_Paulo, which is UTC-3:
| Minute |
Average of CPUUtilization
|
What it was |
|---|---|---|
| 12:00 | 0.0030% | idle, before the load |
| 12:01 | 0.2164% | idle, before the load |
| 12:02 | 15.0042% | the load started inside this minute |
| 12:03 | 100.0% | first breaching data point |
| 12:04 | 99.9967% | second breaching data point |
| 12:05 | 100.0% | third breaching data point |
The alarm went from OK to ALARM at 12:06:06.831, read from describe-alarm-history. The 12:02 data point is the whole story: the load started inside that minute, so the period is partial and its Average came out at 15.0042%, which is not breaching. The three breaching data points are the three full minutes after it, and the transition lands in the minute after the last of them. The 180-second floor assumes the load starts aligned with a period boundary, and nothing aligns with a period boundary. Plan for the extra partial minute.
The 65 seconds the arithmetic does not predict are the cost of reading an aggregate, not latency in the path. Once the alarm transitioned, the Automation execution started 274 milliseconds later.
It is also a property of the design rather than a defect: remediation that fires on the first breaching data point reacts to noise.
When the alarm does not transition and the configuration looks right, there is a published runbook for that. AWSSupport-TroubleshootCloudWatchAlarm exists to identify and diagnose misconfigured or problematic CloudWatch alarms, detecting delayed or missing data points that can lead to missed or delayed alarm actions.
A metric alarm cannot invoke Run Command as an alarm action
The actions supported for a metric alarm are listed in the documentation: notify subscribers through an SNS topic, invoke a Lambda function, perform an EC2 action such as stopping, terminating, rebooting or recovering an instance, perform an Auto Scaling action, create an OpsItem in Systems Manager Ops Center or an incident in Incident Manager, and start an investigation. The same page adds that alarms also emit events to EventBridge when they change state, and that EventBridge can be set up to trigger other actions from those state changes.
Run Command is not on that list. Automation is not on that list. So the alarm cannot reach the node by itself, and there are exactly two documented ways to cross that gap: the EventBridge hop this post builds, or a Lambda function invoked as the alarm action, which then calls SendCommand with code you wrote. The EventBridge hop is not an extra hop that could be removed; it is the only documented path from a metric alarm to an execution on the node that requires no code of your own. Anyone looking at the diagram and asking why the alarm does not call Systems Manager directly is asking the right question, and the answer is that the direct edge does not exist, while the two indirect ones differ in what they ask you to own. The design section above argues the Lambda version on its merits rather than pretending it is unavailable.
What else exists nearby: creating an OpsItem from the alarm is a native action, and it is the right one for opening the operational item. It does not execute a procedure.
The CloudFormation template for the Automation runbook path
Full file: iac/ec2-remediation-automation-runbook.yaml. It builds its own VPC, a workload node registered as a managed node, the two Systems Manager documents, the three IAM roles, the two rules, the dead letter queue, the encryption key, the alert topic, the log group with its metric filter, and six CloudWatch alarms counting the CPU alarm itself. The excerpts below are the parts where a decision had to be made.
The header and the parameters that carry a real constraint:
AWSTemplateFormatVersion: '2010-09-09'
Description: CPU remediation lab where a CloudWatch alarm starts a Systems Manager Automation runbook through an EventBridge rule, with a dead letter queue, encryption at rest and alarms.
Parameters:
WorkloadAmiId:
Type: 'AWS::SSM::Parameter::Value<AWS::EC2::Image::Id>'
Default: /aws/service/ami-amazon-linux-latest/al2023-ami-kernel-default-x86_64
Description: Public SSM parameter that resolves to the latest Amazon Linux 2023 AMI, which ships the SSM Agent preinstalled.
InstanceType:
Type: String
Default: t3.micro
AllowedValues: [t3.nano, t3.micro]
Description: Instance type for the workload node. Kept at nano or micro so the lab stays inside the account guard rail.
ConstraintDescription: Must be t3.nano or t3.micro.
VpcCidr:
Type: String
Default: 10.20.0.0/16
AllowedPattern: '^(\d{1,3}\.){3}\d{1,3}/\d{1,2}$'
Description: CIDR block for the lab VPC. The lab never touches a pre-existing VPC.
ConstraintDescription: Must be a valid IPv4 CIDR block.
PublicSubnetCidr:
Type: String
Default: 10.20.1.0/24
AllowedPattern: '^(\d{1,3}\.){3}\d{1,3}/\d{1,2}$'
Description: CIDR block for the public subnet that hosts the workload node.
ConstraintDescription: Must be a valid IPv4 CIDR block.
EnableDetailedMonitoring:
Type: String
Default: 'true'
AllowedValues: ['true', 'false']
Description: Enables EC2 detailed monitoring, which is what publishes CPUUtilization in 1-minute periods. A 60-second alarm period is only honest with this set to true.
CpuThresholdPercent:
Type: Number
Default: 90
MinValue: 1
MaxValue: 100
Description: CPU utilization percentage above which the alarm breaches.
AlarmPeriodSeconds:
Type: Number
Default: 60
AllowedValues: [60, 300]
Description: Alarm period in seconds. Only the two values that match a real EC2 publishing interval are allowed, 60 with detailed monitoring and 300 without it.
ConstraintDescription: Must be 60 or 300.
AlarmPeriodSeconds uses AllowedValues rather than a free integer on purpose. The alarm resource accepts 10, 20, 30, 60 and any multiple of 60, but EC2 only publishes CPUUtilization at 5-minute or 1-minute intervals. A 10-second period over a metric published once a minute produces an alarm that sits without data most of the time. The two allowed values are the two that correspond to a real publishing configuration, and 60 is only honest with EnableDetailedMonitoring set to true. That is a domain restriction AllowedValues can enforce and a Description cannot.
WorkloadTagValue, further down the file, exists because it is the IAM scope. It is the tag that ssm:resourceTag/Workload restricts ssm:SendCommand against, so changing the parameter changes who can be remediated. The parameter is a prefix and not the tag value: every point of use writes !Sub '${WorkloadTagValue}-${AWS::StackName}', on the instance tag, in the IAM condition and, in the second template, in the target set. A parameter Default cannot hold a function, so the separation has to live at the point of use, the same place the metric namespace separation ended up. Without it, two stacks from this template share one tag value, and a tag is an account-wide and Region-wide selector, so one stack's role would be allowed to command the other stack's node. And AllowedValues: [t3.nano, t3.micro] reproduces in the template the ceiling the account guard rail already imposes, so the error surfaces during parameter validation instead of as an AccessDenied inside a CREATE_FAILED.
AllowedValues closes one parameter at a time, and two of the combinations that matter here live between parameters. AlarmDatapointsToAlarm must not exceed AlarmEvaluationPeriods, or CloudWatch refuses the alarm and the stack ends in CREATE_FAILED after it has already built a VPC, an instance and three roles. And a 60-second period with EnableDetailedMonitoring set to false is a combination CloudFormation accepts and the design does not: basic monitoring publishes every 5 minutes, so the alarm would sit without data most of the time and, with TreatMissingData: missing, never transition. Nothing would be broken, and nothing would be remediated either. A Description that warns about this is documentation, not validation, and the native mechanism for an assertion across parameters is the Rules section:
Rules:
PeriodMatchesMonitoringMode:
Assertions:
- Assert: !Not
- !And
- !Equals [!Ref AlarmPeriodSeconds, '60']
- !Equals [!Ref EnableDetailedMonitoring, 'false']
AssertDescription: A 60-second alarm period requires EnableDetailedMonitoring set to true. With basic monitoring EC2 publishes CPUUtilization every 5 minutes, and a 60-second period would leave the alarm without data most of the time.
DatapointsFitOneEvaluationPeriod:
RuleCondition: !Equals [!Ref AlarmEvaluationPeriods, '1']
Assertions:
- Assert: !Equals [!Ref AlarmDatapointsToAlarm, '1']
AssertDescription: With AlarmEvaluationPeriods set to 1, AlarmDatapointsToAlarm must also be 1.
DatapointsFitTwoEvaluationPeriods:
RuleCondition: !Equals [!Ref AlarmEvaluationPeriods, '2']
Assertions:
- Assert: !Or
- !Equals [!Ref AlarmDatapointsToAlarm, '1']
- !Equals [!Ref AlarmDatapointsToAlarm, '2']
AssertDescription: With AlarmEvaluationPeriods set to 2, AlarmDatapointsToAlarm must be 1 or 2.
DatapointsFitThreeEvaluationPeriods:
RuleCondition: !Equals [!Ref AlarmEvaluationPeriods, '3']
Assertions:
- Assert: !Not [!Equals [!Ref AlarmDatapointsToAlarm, '5']]
AssertDescription: With AlarmEvaluationPeriods set to 3, AlarmDatapointsToAlarm cannot be 5.
The shape of those rules is dictated by what Rules can express. The section offers equality, set membership, boolean composition and RuleCondition, and it offers no numeric comparison, so "M must not exceed N" cannot be written as a comparison. The way out is to narrow both counters to AllowedValues: [1, 2, 3, 5] and assert the relation value by value, which is three rules instead of one line. Both templates carry the same four rules. A failed assertion rejects the stack operation before any resource is created, which is the difference between a parameter mistake and a half-built lab.
Conditions:
CreateAlertSubscription: !Not [!Equals [!Ref AlertEmail, '']]
# Monitoring on AWS::EC2::Instance is a Boolean and the parameter is a String.
UseDetailedMonitoring: !Equals [!Ref EnableDetailedMonitoring, 'true']
The second condition exists because Monitoring on AWS::EC2::Instance is a Boolean while the parameter is a String. Passing the string straight through is the kind of thing cfn-lint flags, so the instance uses Monitoring: !If [UseDetailedMonitoring, true, false].
The network comes from the stack rather than from the default VPC, which keeps teardown total and keeps the lab from touching anything pre-existing. There is no NAT Gateway. The security group is where the connectivity decision becomes visible:
WorkloadSecurityGroup:
Type: AWS::EC2::SecurityGroup
Properties:
GroupDescription: Workload node for the CPU remediation lab. Egress only.
VpcId: !Ref LabVpc
# No ingress rule, on purpose, and port 22 appears nowhere: the SSM Agent
# opens every connection outbound, and shell access is Session Manager.
SecurityGroupEgress:
- IpProtocol: tcp
FromPort: 443
ToPort: 443
CidrIp: 0.0.0.0/0
Description: HTTPS to the Systems Manager regional endpoints.
Tags:
- Key: Name
Value: !Sub '${AWS::StackName}-workload-sg'
- Key: Project
Value: !Ref ProjectTag
- Key: Source
Value: post-engine
- Key: Stack
Value: !Ref AWS::StackName
The Systems Manager documentation presents two ways to give a node connectivity: interface endpoints for com.amazonaws.<region>.ssm, .ec2messages and .ssmmessages, or outbound HTTPS on port 443 to the three corresponding regional endpoints. Interface endpoints are the better security posture and carry an hourly charge per availability zone, which puts three of them outside the cost ceiling of a throwaway lab. This template uses a public subnet with outbound 443. Connectivity is the single most common reason a node never registers, so the post says which path it takes and why. And the sentence that closes the security argument: the SSM Agent initiates all connections to the service, so there is no inbound rule to open and port 22 appears nowhere in the file.
The instance installs the load generator and deliberately leaves it stopped:
WorkloadInstance:
Type: AWS::EC2::Instance
# The default route alone does not give the instance a way out: it has to sit
# in the route table that the subnet is associated with. Depending on both
# keeps the node from booting without a path to the Systems Manager
# endpoints, which is the usual reason a node never registers.
DependsOn:
- PublicDefaultRoute
- PublicSubnetRouteAssociation
Properties:
ImageId: !Ref WorkloadAmiId
InstanceType: !Ref InstanceType
IamInstanceProfile: !Ref WorkloadInstanceProfile
SubnetId: !Ref PublicSubnet
SecurityGroupIds:
- !Ref WorkloadSecurityGroup
Monitoring: !If [UseDetailedMonitoring, true, false]
MetadataOptions:
HttpEndpoint: enabled
# IMDSv2 only. The recommended managed node path, Default Host
# Management Configuration, does not support IMDSv1.
HttpTokens: required
BlockDeviceMappings:
- DeviceName: /dev/xvda
Ebs:
VolumeType: gp3
VolumeSize: 8
Encrypted: true
DeleteOnTermination: true
UserData:
Fn::Base64: |
#!/bin/bash
set -euo pipefail
# One busy loop per vCPU. A single loop does not saturate a two-vCPU
# instance, so the alarm would never cross the threshold.
cat > /usr/local/bin/cpu-load-test.sh <<'LOADEOF'
#!/bin/sh
set -u
cpus=$(nproc)
i=0
while [ "${i}" -lt "${cpus}" ]; do
sh -c 'while : ; do : ; done' &
i=$((i + 1))
done
wait
LOADEOF
chmod 0755 /usr/local/bin/cpu-load-test.sh
cat > /etc/systemd/system/cpu-load-test.service <<'UNITEOF'
[Unit]
Description=Synthetic CPU load for the remediation lab
[Service]
Type=simple
ExecStart=/usr/local/bin/cpu-load-test.sh
Restart=no
CPUAccounting=yes
KillMode=control-group
UNITEOF
systemctl daemon-reload
# The unit is installed and deliberately left stopped and disabled.
Tags:
- Key: Name
Value: !Sub '${AWS::StackName}-workload'
# The stack name is part of the tag value because Run Command resolves a
# tag target set across the whole account and Region. A bare parameter
# default would make two stacks share one target set.
- Key: Workload
Value: !Sub '${WorkloadTagValue}-${AWS::StackName}'
- Key: Project
Value: !Ref ProjectTag
- Key: Source
Value: post-engine
- Key: Stack
Value: !Ref AWS::StackName
HttpTokens: required is not decoration. Default Host Management Configuration, which is the recommended way to grant managed node permissions today, requires IMDSv2 and does not support IMDSv1. Even though this template uses an instance profile, keeping IMDSv2 mandatory is what keeps the instance compatible with the recommended path.
Five details of that UserData are decisions rather than noise. The load has to be one busy loop per vCPU, because a single loop does not saturate a two-vCPU instance and the alarm would never cross 90 percent. Those loops are spawned by a small script instead of several ExecStart lines, because systemd only permits multiple ExecStart entries for Type=oneshot, which would run them sequentially and defeat the point. Restart=no, because a unit that restarts itself would defeat the remediation and the experiment would be measuring systemd instead of the design. KillMode=control-group is the systemd default and is written out anyway, because it is what guarantees that systemctl stop kills the forked child loops and not only the parent script. CPUAccounting=yes puts the load in its own cgroup, which gives a second way to identify it beyond the unit name.
The SSM Agent comes preinstalled on AWS-provided Amazon Linux 2023 AMIs, so there is no agent installation step. The load script uses nothing but POSIX shell, so the node needs no package repository.
The encryption key, and why the dead letter queue cannot do without it
This is the finding worth the most in the template, and it comes from reading two pages next to each other.
First: SqsManagedSseEnabled enables server-side encryption using SQS-owned encryption keys, and when the property is not set, SSE-SQS encryption is enabled by default. Second: EventBridge does not support Amazon SQS queues encrypted with an AWS owned key, and that statement covers both targets and queues specified as a target's dead letter queue.
Put them together and an AWS::SQS::Queue declared without saying anything about encryption, used as the dead letter queue of an EventBridge target, is encrypted by default with a key EventBridge does not support. The outcome is the worst available: the invocation failure happens, the message does not reach the queue, and the record of the failure is precisely what is lost. The documented symptom is InvocationsFailedToBeSentToDlq, a metric that exists to count invocations that could not be moved to the dead letter queue because of a permission error, an unavailable resource or a size limit.
Being exact about the status of that conclusion: it is an inference from two documented statements, not a single documented statement. The lab did not close it, and the reason is worth naming. Nothing in the path failed, so EventBridge never attempted a write to the queue. InvocationsFailedToBeSentToDlq stayed at 0 and the queue stayed at zero messages, which is consistent with the inference and evidence for nothing. Closing it means deploying the unsupported encryption on purpose and watching the record of a failure disappear, and that experiment is not in this run.
The fix in the template is a customer managed key, and that choice drags consequences the post declares rather than hides:
LabKey:
Type: AWS::KMS::Key
DeletionPolicy: Delete
UpdateReplacePolicy: Delete
Properties:
Description: !Sub 'Customer managed key for the ${AWS::StackName} remediation lab queue and topic.'
EnableKeyRotation: true
# Shortest waiting period KMS allows. Removing the key from the stack
# schedules deletion and the wait cannot be canceled from CloudFormation.
PendingWindowInDays: 7
KeyPolicy:
Version: '2012-10-17'
Statement:
- Sid: AllowAccountIamPolicies
Effect: Allow
Principal:
AWS: !Sub 'arn:${AWS::Partition}:iam::${AWS::AccountId}:root'
Action: 'kms:*'
Resource: '*'
# The two service statements are asymmetric, and the asymmetry is the
# documentation and not a preference. The SNS guidance on encrypted
# topics instructs adding aws:SourceAccount and aws:SourceArn to the
# key policy, and then carves out one exception: those keys are not
# supported for EventBridge-to-encrypted-topic delivery. So the
# EventBridge statement goes without them and the CloudWatch
# statement carries them, scoped to the alarms of this stack.
# GenerateDataKey carries a wildcard because the SNS documentation
# requires kms:GenerateDataKey* for a service publishing to an
# encrypted topic: the service may call either GenerateDataKey or
# GenerateDataKeyWithoutPlaintext.
- Sid: AllowEventBridgeToUseTheKey
Effect: Allow
Principal:
Service: events.amazonaws.com
Action:
- kms:Decrypt
- 'kms:GenerateDataKey*'
Resource: '*'
- Sid: AllowCloudWatchAlarmsToPublish
Effect: Allow
Principal:
Service: cloudwatch.amazonaws.com
Action:
- kms:Decrypt
- 'kms:GenerateDataKey*'
Resource: '*'
Condition:
StringEquals:
aws:SourceAccount: !Ref AWS::AccountId
ArnLike:
aws:SourceArn: !Sub 'arn:${AWS::Partition}:cloudwatch:${AWS::Region}:${AWS::AccountId}:alarm:${AWS::StackName}-*'
Four things about that key belong in the text. One key serves two grantees, the dead letter queue and the SNS topic, so the key policy accumulates the grants of both publishing services, and those grants come from two different pages that have to be read together. The EventBridge targets page names kms:Decrypt and kms:GenerateDataKey for writing to an encrypted target, and its example is an SQS queue, which is the dead letter path here. The SNS page on encryption keys covers the other path, a service publishing to an encrypted topic, and it requires kms:GenerateDataKey* with the wildcard, because the publisher may call GenerateDataKey or GenerateDataKeyWithoutPlaintext. The wildcard satisfies both pages at once, so both statements carry it, and it is not a convenience: two service principals publish to this topic, events.amazonaws.com for the failure notification rule and cloudwatch.amazonaws.com for the five operational alarms, and a permission covering only one of the two variants of the call would fail by producing silence, which is indistinguishable from nothing having gone wrong. The second thing is the opening statement, which grants the account root kms:* and is what keeps the key governable; without it the key becomes unmanageable.
The third thing is the only asymmetry in the template, and it is the fact rather than a preference: of the two service statements on the key, one names its source and the other cannot. The SNS guidance on encrypted topics instructs adding aws:SourceAccount and aws:SourceArn to the key policy to close the confused deputy opening, and publishes the example carrying both. Then it carves out one exception, and the exception names one product: those condition keys are not supported for EventBridge-to-encrypted-topic delivery. So the cloudwatch.amazonaws.com statement carries both, scoped by ArnLike to the alarms whose names start with this stack's name, and the events.amazonaws.com statement carries neither, because the page that asks for them is the page that rules them out for that path, and the EventBridge example for a target using a customer managed key publishes exactly that statement without a condition. A condition that never matches inside an Allow is worse than no condition: the statement becomes dead letters, and here the thing that would die is the dead letter write and the failure notification, which are the two mechanisms this design counts on when something goes wrong. What limits the remaining exposure is that permission on the key is not permission on the data. Writing to this queue needs sqs:SendMessage, which the queue policy grants only to the rules of this stack in this account, and publishing to this topic needs sns:Publish, which the topic policy grants to one rule by exact ARN and to the alarms by name prefix. Whether the source-scoped form would still permit the dead letter write on the EventBridge statement is a row in the results matrix, not something I am going to assert without applying it.
The scope on the cloudwatch.amazonaws.com statement deserves the same treatment, because that argument cuts both ways. The instruction to add both condition keys to the key policy belongs to the SNS page, and the same page says to check each service's own documentation for the exact details of each case. What the CloudWatch page documents is aws:SourceArn as the ARN of the alarm on the topic policy, for sns:Publish. Neither page states that CloudWatch populates aws:SourceArn on the GenerateDataKey call against the key. aws:SourceAccount is the half with less to lose; the ArnLike next to it is a bet on the generic instruction being literal for this call too. If that condition key is absent from that call, the condition never matches, and the five operational alarms stop publishing with the same silence the paragraph above objects to. So that is a second row in the results matrix, and it was not exercised either.
The fourth thing is PendingWindowInDays: 7, and it is the teardown story: removing the key from the stack schedules deletion and starts the mandatory waiting period, which ranges from 7 to 30 days and defaults to 30, during which the key sits in Pending Deletion and CloudFormation cannot cancel the deletion. delete-stack completes and the key keeps existing for seven days.
The waiting period is the part people expect to cost money, and it does not. The pricing page states there is no charge for a customer managed key that is scheduled for deletion, and that cancelling the deletion during the waiting period restores charges as though it had never been scheduled. The seven days are a window where the key is unusable for cryptographic operations and free. What costs money is the key existing in the first place: each key you create is billed at USD 1 per month, prorated hourly. Shortening PendingWindowInDays from the 30-day default to 7 therefore buys a faster key quota slot and a shorter window in which someone could cancel the deletion, not a smaller bill.
The queue itself, and the policy that lets this stack's rules and nothing else write to it:
RemediationDlq:
Type: AWS::SQS::Queue
DeletionPolicy: Delete
UpdateReplacePolicy: Delete
Properties:
MessageRetentionPeriod: 1209600
KmsMasterKeyId: !Ref LabKey
KmsDataKeyReusePeriodSeconds: 300
Tags:
- Key: Project
Value: !Ref ProjectTag
- Key: Source
Value: post-engine
- Key: Stack
Value: !Ref AWS::StackName
RemediationDlqPolicy:
Type: AWS::SQS::QueuePolicy
Properties:
Queues:
- !Ref RemediationDlq
PolicyDocument:
Version: '2012-10-17'
Statement:
- Sid: AllowOnlyThisStacksRulesToDeadLetter
Effect: Allow
Principal:
Service: events.amazonaws.com
Action: sqs:SendMessage
Resource: !GetAtt RemediationDlq.Arn
# A rule name prefix and not an exact ARN, because both rules of
# this stack dead letter here, and one statement naming the stack's
# rule namespace says that more plainly than two statements.
Condition:
ArnLike:
aws:SourceArn: !Sub 'arn:${AWS::Partition}:events:${AWS::Region}:${AWS::AccountId}:rule/${AWS::StackName}-*'
StringEquals:
aws:SourceAccount: !Ref AWS::AccountId
aws:SourceArn and aws:SourceAccount are populated here because EventBridge, an AWS service, is calling sqs:SendMessage on behalf of a rule. Those two conditions are what closes the confused deputy opening on this queue: without them, the statement grants sqs:SendMessage to the EventBridge service principal for any rule in any account, which includes rules that are not yours. The scope is a rule name prefix rather than one exact ARN, and the reason is that two rules in this stack send failed invocations to this queue, the remediation rule and the execution failure rule described further down. Both rules are named from the stack name, so the prefix covers them and covers nothing else in the account. The same pair of keys appears on the topic policy for the same reason, exact there for the rule and prefixed for the alarms. On the key policy only one of the two statements can carry them, and the section above says why.
The two Systems Manager documents
The command document declares no parameters, and that absence is the security design:
RemediationCommandDocument:
Type: AWS::SSM::Document
Properties:
DocumentType: Command
DocumentFormat: YAML
TargetType: /AWS::EC2::Instance
UpdateMethod: Replace
# No parameters, on purpose: nothing from the event reaches the shell, so
# there is no injection surface to defend.
Content:
schemaVersion: '2.2'
description: Stops the synthetic CPU load unit and reports whether the condition cleared.
mainSteps:
- action: aws:runShellScript
name: stopTestLoad
inputs:
timeoutSeconds: '120'
runCommand:
- 'set -u'
- 'UNIT="cpu-load-test.service"'
- 'echo "REMEDIATION_START unit=${UNIT} instance=$(cat /var/lib/cloud/data/instance-id 2>/dev/null) at=$(date -u +%FT%TZ)"'
- ''
- 'if systemctl is-active --quiet "${UNIT}"; then'
- ' systemctl stop "${UNIT}"'
- ' echo "REMEDIATION_ACTION stopped=${UNIT}"'
- 'else'
- ' echo "REMEDIATION_ACTION already_inactive=${UNIT}"'
- 'fi'
- ''
- 'read -r _ u n s i rest < /proc/stat'
- 'busy1=$((u + n + s)); idle1=${i}'
- 'sleep 5'
- 'read -r _ u n s i rest < /proc/stat'
- 'busy2=$((u + n + s)); idle2=${i}'
- ''
- 'dbusy=$((busy2 - busy1)); didle=$((idle2 - idle1)); dtotal=$((dbusy + didle))'
- 'used=0'
- '[ "${dtotal}" -gt 0 ] && used=$(( (dbusy * 100) / dtotal ))'
- ''
- 'if [ "${used}" -ge 90 ]; then'
- ' echo "REMEDIATION_FAILED cpu_still_high=${used}"'
- ' exit 1'
- 'fi'
- ''
- 'echo "REMEDIATION_OK cpu=${used}"'
Every line of that script answers a requirement. systemctl stop on an already stopped unit returns success, which gives idempotency with no extra conditional. The measurement comes from deltas of /proc/stat, which exists without installing anything and works on a minimal AMI. The three markers, REMEDIATION_OK, REMEDIATION_FAILED and REMEDIATION_ACTION, are the contract with the metric filter. And the non-zero exit code is what makes the Automation step fail when the condition persists, instead of reporting success for a remediation that did not remediate. That distinction gets its own section later.
There is a parameter injection surface in the general case, and the document closes it by not having parameters. For documents where an outside value genuinely has to reach a string parameter, the plugin reference offers interpolationType, which treats parameter inputs as string literals rather than potentially executable commands. This design does not need the mechanism because it eliminates the parameter, and saying that is stronger than using it.
The runbook is where the event value lands:
RemediationRunbook:
Type: AWS::SSM::Document
Properties:
DocumentType: Automation
DocumentFormat: YAML
UpdateMethod: Replace
Content:
schemaVersion: '0.3'
description: Runs the CPU remediation command document on the instance that breached the alarm.
assumeRole: '{{ AutomationAssumeRole }}'
parameters:
AutomationAssumeRole:
type: String
description: Execution role the Automation service assumes.
default: !GetAtt AutomationExecutionRole.Arn
InstanceId:
type: String
description: Identifier of the breaching instance, supplied by the EventBridge input transformer.
allowedPattern: '^i-[0-9a-f]{8,17}$'
AlarmName:
type: String
description: Name of the alarm that transitioned, carried into the command comment for traceability.
maxChars: 100
mainSteps:
- name: runRemediation
action: 'aws:runCommand'
onFailure: Abort
inputs:
DocumentName: !Ref RemediationCommandDocument
InstanceIds:
- '{{ InstanceId }}'
Comment: '{{ AlarmName }}'
# Delivery timeout, not execution timeout: this bounds the wait
# for the SSM Agent to receive the command. The script duration
# ceiling is timeoutSeconds inside the command document.
TimeoutSeconds: 120
CloudWatchOutputConfig:
CloudWatchLogGroupName: !Ref CommandOutputLogGroup
CloudWatchOutputEnabled: true
outputs:
- 'runRemediation.CommandId'
- 'runRemediation.Status'
- 'runRemediation.ResponseCode'
- 'runRemediation.Output'
Five notes on this block.
Neither document is given a Name, so CloudFormation generates one and !Ref returns it. That is not laziness: document names must match ^[a-zA-Z0-9_\-.]{3,128}$, and aws, amazon, amzn, AWSEC2, AWSConfigRemediation and AWSSupport are reserved prefixes. Letting CloudFormation name the document removes a class of deploy failure that depends on what someone called the stack.
The shape of the aws:runCommand step follows the YAML example in the resource reference, with InstanceIds as an input of the step, a sibling of DocumentName and Parameters. The JSON example on the same page puts InstanceIds inside Parameters, which does not match the action reference. Two forms on one page, and only one of them agrees with the action documentation.
Comment carries the alarm name and never the shell. The value that comes from the event enters an API field, not a command argument, which is the other half of why there is no injection surface here. The identifier is also validated on the way in: InstanceId declares allowedPattern: '^i-[0-9a-f]{8,17}$', which is the only format check on an event-supplied value anywhere in this design, and which is what the parameter security guidance for SSM documents asks for, input validation restricting a parameter to a safe pattern. A transformer that resolves to something other than an instance identifier is rejected on the parameter. The documentation does not state the point at which that rejection lands, and nothing here was executed, so the post claims the check and not its timing. onFailure: Abort on the step is written out rather than accepted silently as a default.
Two length ceilings sit on the alarm name, and only one of them can be reached. SendCommand caps the comment at 100 characters, so the alarm name goes in and nothing else, and the runbook parameter declares maxChars: 100 to say that inside the document. That declaration gets the same hedge the pattern above it got: the length property appears in a published example of a command document parameter and not in the published example of the schema this runbook uses, so it is there for the same reason the pattern is, and nothing here was executed. The alarm is named ${AWS::StackName}-workload-cpu-high, which spends 18 of those 100 characters before the operator types anything, so a stack name above 82 would produce an alarm name longer than the comment field accepts. In this template that case cannot be reached, and the reason is a tighter ceiling one layer up: Name on an EventBridge rule is capped at 64 characters, the longer suffix here is -remediation-failed at 19, so a stack name above 45 characters fails the creation of the rule and the stack rolls back before any alarm is armed. A CREATE_FAILED naming the rule is a better diagnosis than an execution rejected on a parameter, which is the ordering worth keeping in mind when a template chooses to name its own resources. The length declaration stays in the runbook because it is the ceiling that would bite a template that let CloudFormation name its rules instead.
The two timeouts in this design are different things, and the names do not help. TimeoutSeconds on the aws:runCommand step, valid from 30 to 2592000, is the time to wait for the command to be delivered to the SSM Agent; if the agent has not received it by then, the command status becomes Delivery Timed Out. The ceiling on how long the script may run is timeoutSeconds inside the command document, set to 120 there. A script that hangs is bounded by the second one, and an agent that is not listening is caught by the first.
The alarm, the rule, and the transformer
CpuUtilizationAlarm:
Type: AWS::CloudWatch::Alarm
Properties:
AlarmName: !Sub '${AWS::StackName}-workload-cpu-high'
AlarmDescription: CPU utilization on the workload node stayed above the threshold long enough to warrant the remediation procedure.
Namespace: AWS/EC2
MetricName: CPUUtilization
Dimensions:
- Name: InstanceId
Value: !Ref WorkloadInstance
Statistic: Average
# No Unit. The CloudWatch API reference recommends omitting it, because
# naming a unit the metric does not publish is what strands an alarm in
# INSUFFICIENT_DATA.
Period: !Ref AlarmPeriodSeconds
EvaluationPeriods: !Ref AlarmEvaluationPeriods
DatapointsToAlarm: !Ref AlarmDatapointsToAlarm
Threshold: !Ref CpuThresholdPercent
ComparisonOperator: GreaterThanThreshold
# Declared, not inherited. With breaching, a node that stops publishing
# would enter ALARM and remediation would fire at a silent node.
TreatMissingData: missing
ActionsEnabled: true
# No AlarmActions: the state change event reaches EventBridge regardless.
Tags:
- Key: Project
Value: !Ref ProjectTag
- Key: Source
Value: post-engine
- Key: Stack
Value: !Ref AWS::StackName
TreatMissingData: missing is the documented default and is declared anyway, for a specific reason: with breaching, an instance that stops publishing metrics would enter ALARM and remediation would fire against a node that may not be responding. Remediating the absence of data is the opposite of the intent.
There is no Unit on the alarm, and the reason runs against a common assumption. Percent is the documented unit of CPUUtilization, so naming it would work. But the PutMetricAlarm reference recommends omitting the unit so that you do not inadvertently specify a unit that is not published for the metric, and it is naming the wrong unit that leaves the alarm stuck in INSUFFICIENT DATA. Omitting it makes CloudWatch retrieve every unit published for the metric and evaluate the alarm, which works as intended for a metric published with a single unit. The failure runs the opposite way from the way most people assume, and the documented recommendation is the one worth copying into a template someone else will adapt.
There are no AlarmActions on this alarm. The alarm needs no configured action for the EventBridge rule to fire, because CloudWatch sends the state change event either way. If an operator wants a human notification, the alarm action is additive and independent of the remediation path.
CpuRemediationRule:
Type: AWS::Events::Rule
Properties:
Name: !Sub '${AWS::StackName}-cpu-remediation'
Description: Starts the remediation runbook when the CPU alarm moves from OK to ALARM.
State: !Ref RuleState
EventPattern:
source:
- aws.cloudwatch
detail-type:
- CloudWatch Alarm State Change
detail:
alarmName:
- !Ref CpuUtilizationAlarm
state:
value:
- ALARM
previousState:
value:
- OK
Targets:
- Id: StartRemediationRunbook
Arn: !Sub 'arn:${AWS::Partition}:ssm:${AWS::Region}:${AWS::AccountId}:automation-definition/${RemediationRunbook}:$DEFAULT'
RoleArn: !GetAtt AlarmInvocationRole.Arn
InputTransformer:
InputPathsMap:
instanceId: '$.detail.configuration.metrics[0].metricStat.metric.dimensions.InstanceId'
alarmName: '$.detail.alarmName'
# No quotes around the variables: EventBridge adds them for values
# that represent strings. Quotes are permitted here, and what the
# documentation rules out is quoting a variable that represents a
# JSON object or an array.
InputTemplate: '{"InstanceId":[<instanceId>],"AlarmName":[<alarmName>]}'
RetryPolicy:
MaximumRetryAttempts: 2
# Five minutes. The alarm needs three minutes of breaching data
# points to transition, so an event older than that describes a
# window that has already closed, and not acting beats acting on
# stale information.
MaximumEventAgeInSeconds: 300
DeadLetterConfig:
Arn: !GetAtt RemediationDlq.Arn
Ref on an AWS::CloudWatch::Alarm returns the alarm name, which is exactly what detail.alarmName carries, so the pattern and the alarm cannot drift apart.
previousState.value matches only OK, and the cost of that is explicit. A newly created alarm starts in INSUFFICIENT_DATA. An instance coming up has high boot CPU, and an INSUFFICIENT_DATA to ALARM transition in the first minutes of its life would remediate a condition that is only initialization. Including INSUFFICIENT_DATA in the pattern widens coverage and creates that false positive. The choice here is narrower, predictable coverage, and the cost is that an alarm going from INSUFFICIENT_DATA straight to ALARM for any reason will not remediate. Anyone who wants that coverage safely should look at WarmUpConfiguration on the alarm resource, which delays evaluation after the alarm is created or updated, keeping it in INSUFFICIENT_DATA without executing actions during the period. It is not in this template.
The target ARN ends in a version selector. The documented ARN form for an automation definition is automation-definition/<name>:<version>, and the final segment selects the document version. This template uses $DEFAULT there, which is the version-agnostic form an operator wants, and which I could not confirm in a reference page. That is another row the results matrix leaves open: the stack carrying this ARN reached CREATE_COMPLETE, which is acceptance at create time and says nothing about how the selector resolved when the rule fired. If the service rejects it, the replacement is :1, which is the correct version for a document created with the default UpdateMethod: Replace. The $DEFAULT string passes through Fn::Sub untouched, because Fn::Sub only interprets ${.
The InputTemplate has no quotes around the variables, and the documentation says two different things about that. Its list of common issues states that quotes are not required for variables that represent strings, that they are permitted, and that EventBridge adds them during transformation to keep the output valid JSON; what the page rules out is quoting a variable that represents a JSON object or an array. The note in the input path section goes the other way and instructs putting quotes around variables that refer to JSON path parameters. The unquoted form is the one the page's own Simple JSON example publishes and the one the official walkthrough uses, so it is the one here. The bracket form, {"InstanceId":[<instanceId>]}, is there because Parameters on StartAutomationExecution is a map of string to list of strings.
AutomationAssumeRole is deliberately absent from the InputTemplate. It is the default of the parameter inside the runbook, resolved by !GetAtt AutomationExecutionRole.Arn. That keeps the transformer at two variables and keeps an ARN out of a hand-assembled JSON string.
MaximumEventAgeInSeconds: 300 is a decision, not a default, and the number comes from the alarm. The transition needs three minutes of breaching data points, so five minutes is the alarm's own evaluation window plus a margin. An event older than that describes a window that has already closed, and limiting the age means preferring not to act over acting on stale information. The property accepts anything from 60 to 86400 seconds, and a day-old remediation event is the argument for not leaving it at the default.
The failure notification rule and the observability resources round out the file:
RemediationFailureRule:
Type: AWS::Events::Rule
Properties:
Name: !Sub '${AWS::StackName}-remediation-failed'
Description: Notifies the alert topic when a remediation automation ends in a terminal state other than success.
State: !Ref RuleState
EventPattern:
source:
- aws.ssm
detail-type:
- EC2 Automation Execution Status-change Notification
detail:
Definition:
- !Ref RemediationRunbook
# Canceled carries one L here. That is the spelling the Systems
# Manager event reference uses for this detail type, and an event
# pattern matches a string exactly, so Cancelled would never match.
Status:
- Failed
- TimedOut
- Canceled
- Rejected
Targets:
- Id: NotifyAlertTopic
Arn: !Ref AlertTopic
# This rule is the only mechanism in the design that tells a person a
# remediation failed, so it gets the same retry and the same dead
# letter queue as the remediation rule. A notification that fails
# permanently with nowhere to land would erase the record of the
# failure it was reporting.
RetryPolicy:
MaximumRetryAttempts: 2
MaximumEventAgeInSeconds: 300
DeadLetterConfig:
Arn: !GetAtt RemediationDlq.Arn
This rule carries the same RetryPolicy and the same DeadLetterConfig as the remediation rule, and the reason is worth more than the symmetry. It is the only thing in the design that tells a person something went wrong. A publish to the topic that fails permanently with no retry and no queue behind it loses the notification and the record of the notification, which is the same defect this post objects to when it talks about a dead letter write that cannot happen, applied one layer up: the failure of the mechanism that reports failures. The template also carries an alarm on FailedInvocations for this rule, and that alarm publishes to the topic the rule could not reach, so it is a trigger for someone to open the console rather than a guarantee that the message arrives. Naming that limit is better than pretending the chain terminates somewhere safe.
The field names come from the published event example, which carries ExecutionId, Definition, DefinitionVersion, Status, StartTime, EndTime and ExecutedBy in detail. The status values come from the EventBridge event reference, which lists, for this detail type, the closed set you can add to a rule: Approved, Canceled, Failed, PendingApproval, PendingChangeCalendarOverride, Rejected, Scheduled, Success and TimedOut. The pattern takes the four that end an execution without success. Approved and Scheduled are not endings, and the two pending states describe an execution waiting on an approval step or on a change calendar. Rejected is in the pattern for the same reason, since a rejected approval also ends the execution, and it costs nothing to match a state this runbook cannot currently reach, because it has no approval step today and adding one should not silently drop a notification.
One character in that list is worth a paragraph of its own, because it is the kind of defect that no tool reports. The documented spelling is Canceled, with one L. An event pattern matches a string exactly, so the British spelling with two L matches nothing the service ever emits: a pattern carrying it looks like it covers that state and does not, cfn-lint passes, the deploy succeeds, the console shows a healthy rule, and the path that quietly stops working is the notification that a remediation failed. It matters more than it sounds because cancellation is not hypothetical in this design. Attaching an alarm to the automation so that a transition to ALARM cancels the execution is in the hardening list further down, and a canceled execution is exactly the event that would go unreported.
ScriptFailureMetricFilter:
Type: AWS::Logs::MetricFilter
Properties:
LogGroupName: !Ref CommandOutputLogGroup
FilterName: !Sub '${AWS::StackName}-remediation-failed-marker'
FilterPattern: '"REMEDIATION_FAILED"'
MetricTransformations:
# No dimensions here, on purpose: a metric filter that assigns
# dimensions cannot also assign a default value, and the default value
# reports a zero for every period in which logs are ingested and
# nothing matches. What keeps the alarm out of INSUFFICIENT_DATA
# between remediations is TreatMissingData on the alarm itself. With no
# dimension available, the stack name goes into the namespace, so two
# stacks do not write to the same series.
- MetricName: RemediationScriptFailures
MetricNamespace: !Sub '${ProjectTag}/EC2Remediation/${AWS::StackName}'
MetricValue: '1'
DefaultValue: 0
MetricTransformations accepts at most one item, and the documentation records a constraint that is easy to miss: if you assign dimensions to the metric a metric filter creates, you cannot assign a default value to it. This filter declares no dimensions and declares DefaultValue: 0. The default value reports a zero for every period in which logs are ingested and nothing matches, which is what separates "the remediation ran and did not report failure" from "the remediation did not run". It does not cover the gaps between remediations: the same documentation page is explicit that if no logs are ingested during a period, no value is reported at all, and in this design the log group only receives lines while a command is running. What keeps the alarm out of INSUFFICIENT_DATA between executions is TreatMissingData: notBreaching on the alarm itself. That trade between dimensions and a default value is real and almost nobody writes about it, and it has a consequence that shows up the moment two stacks exist. With no dimension to separate them, two deployments of this template would count each other's script failures in one metric series, and each stack's alarm would fire on the other stack's node. The separation has to go somewhere, so it goes into the namespace: ${ProjectTag}/EC2Remediation/${AWS::StackName}. The cost is one metric namespace per stack instead of one dimension per stack, which is uglier in a dashboard and correct in an alarm.
The template carries five operational alarms. FailedInvocations and InvocationsFailedToBeSentToDlq in AWS/Events dimensioned on the remediation rule, FailedInvocations again dimensioned on the execution failure rule, ApproximateNumberOfMessagesVisible in AWS/SQS with the QueueName dimension, and RemediationScriptFailures in the custom namespace. That third one has to exist separately because the RuleName dimension ties an alarm to one rule: a second rule without a second alarm has a metric and no reader. All five use TreatMissingData: notBreaching, because the EventBridge metrics are only sent to CloudWatch when they are non-zero, so missing data is the good outcome and treating it as breaching would produce an alarm that is always on.
The CloudFormation template for the direct Run Command path
Full file: iac/ec2-remediation-run-command-target.yaml. Same environment, shorter path. The network, the security group, the instance profile, the instance with its UserData, the command document, the KMS key with its alias, the dead letter queue, the alert topic and the CPU alarm are the same as in the first template, on a different CIDR and under a different stack name so both stacks can run side by side. Two things inside that list differ, and both follow from the shorter path. This stack has a single rule, so the queue policy names that rule's exact ARN instead of the stack's rule name prefix. And the topic policy here carries only the statement for the alarms, because no rule in this stack publishes to the topic, which is the same loss of observability the next paragraph develops. The different CIDR is what keeps the networks apart; what keeps the remediation paths apart is the tag value, and that is a separate problem addressed below.
What leaves is more than the runbook, and the extra losses are the interesting part. Out go the Automation runbook, the Automation execution role and the execution failure rule, which is expected: with no runbook there is no automation execution event to observe. Out also go the log group, the metric filter and the script failure alarm, and that one is not obvious until you look for where the command output would land. CloudWatchOutputConfig is a request parameter of SendCommand. The EventBridge Run Command target exposes RunCommandParameters for the target set and Input for the document and its parameters, and nothing that sets the output configuration, and there is no aws:runCommand step here to carry it. So nothing in this stack can route command output to CloudWatch Logs, which means a metric filter over that log group would never match, the custom metric would sit at its default value forever, and an alarm with TreatMissingData: notBreaching on top of it would stay green by having nothing to read. Shipping those three resources would have been shipping an observability mechanism that structurally cannot fire, so they are not in the file. The inline policy that let the instance write to the log group is gone with them, and so is the LogRetentionInDays parameter.
Where the output lives instead: the Run Command APIs, through get-command-invocation and list-command-invocations, plus the command status events. That is what the direct stack produced, and the last two lines are the point:
$ aws ssm get-command-invocation --command-id 2ef2b3a1-073f-4203-93d1-b1c013e3249b --instance-id i-0c30d903851511f06
Status: Success
ResponseCode: 0
StandardOutputContent: REMEDIATION_START unit=cpu-load-test.service instance=i-0c30d903851511f06 at=2026-10-13T16:57:50Z
REMEDIATION_ACTION stopped=cpu-load-test.service
REMEDIATION_OK cpu=0
CloudWatchOutputConfig: {'CloudWatchLogGroupName': '', 'CloudWatchOutputEnabled': False}
$ aws logs describe-log-groups --log-group-name-prefix /aws/ssm
(no results)
CloudWatchOutputEnabled is False, the log group name is empty, and the Region had no /aws/ssm log group at all while the direct stack was running. The remediation worked and left no log. Everything built on top of a log group, which in the first template is the metric filter and the script failure alarm, has nothing to read in this one.
That is the fourth mark against the direct target, so it is worth naming the first three in one place. First, it cannot aim at the node that breached, because the target is a static list or a tag selector rather than a value lifted from the event. Second, the blast radius of a tag selector is resolved across the account and the Region, not inside the stack. Third, there is no place to put a parameter the event carries, because the integration takes no input transformation path that reaches the document. The fourth is this one: no CloudWatchOutputConfig, so the script output never lands in a log group, which removes the metric filter and the alarm built on it. This is the one I did not expect to find. Three operational alarms remain, on FailedInvocations, on InvocationsFailedToBeSentToDlq and on dead letter queue depth.
What changes is one role and one target. The role:
Policies:
- PolicyName: SendRemediationCommand
PolicyDocument:
Version: '2012-10-17'
Statement:
- Sid: SendOnlyThisDocument
Effect: Allow
Action: ssm:SendCommand
Resource: !Sub 'arn:${AWS::Partition}:ssm:${AWS::Region}:${AWS::AccountId}:document/${RemediationCommandDocument}'
# Adding any global condition key to a SendCommand statement makes
# aws:ViaAWSService mandatory, so this statement deliberately
# carries only the service specific ssm:resourceTag key.
- Sid: SendOnlyToTaggedNodes
Effect: Allow
Action: ssm:SendCommand
Resource: !Sub 'arn:${AWS::Partition}:ec2:${AWS::Region}:${AWS::AccountId}:instance/*'
Condition:
StringEquals:
# Stack scoped, not parameter scoped: the tag target set is
# resolved account wide, so without the stack name this role
# could also command the node of another stack.
ssm:resourceTag/Workload: !Sub '${WorkloadTagValue}-${AWS::StackName}'
ssm:StartAutomationExecution becomes ssm:SendCommand, and iam:PassRole disappears because there is no execution role to pass. Ref on an AWS::SSM::Document returns the document name, which is what the document/<name> ARN needs.
The target:
Targets:
- Id: SendRemediationCommand
Arn: !Sub 'arn:${AWS::Partition}:ssm:${AWS::Region}:${AWS::AccountId}:document/${RemediationCommandDocument}'
RoleArn: !GetAtt AlarmInvocationRole.Arn
# RunCommandParameters does not accept dynamic path parameters, so the
# target set is fixed at deploy time: a tag match, not the node that
# actually breached. The tag value carries the stack name because the
# match is evaluated across the account and Region, not inside this
# stack or inside its VPC.
RunCommandParameters:
RunCommandTargets:
- Key: 'tag:Workload'
Values:
- !Sub '${WorkloadTagValue}-${AWS::StackName}'
RetryPolicy:
MaximumRetryAttempts: 2
# Five minutes. The alarm needs three minutes of breaching data
# points to transition, so an event older than that describes a
# window that has already closed.
MaximumEventAgeInSeconds: 300
DeadLetterConfig:
Arn: !GetAtt RemediationDlq.Arn
There is no InputTransformer and no Input, and the second absence deserves a flag rather than a claim. The targets page says that to specify which SSM document to run and pass parameters to it you use the Input field on the target, as a JSON object with DocumentName, DocumentVersion and Parameters. The AWS CLI example on that same page omits DocumentName from Input and identifies the document through the target Arn, passing only Parameters. This template follows the example: the ARN names the document, and the document declares no parameters, so there is nothing left to pass. Two further notes narrow the gap without closing it, and they do not weigh the same. A note under that CLI example says the Input field passes the document name and parameters to Run Command, that this differs from InputTransformer, which transforms the event payload, and that document parameters belong in Input while target instances belong in RunCommandParameters. The input transformation page states that CloudWatch Logs and SSM targets do not support Input and InputPath in their input transformers, which is the sentence most favorable to what this template does, namely declare neither. What the documentation left open is whether a target carrying no Input at all is accepted. It is: the stack created, the alarm transitioned, and the command was dispatched and succeeded, so the one-line fallback of Input: '{"DocumentName": "<document name>"}' was never needed.
$ aws ssm list-command-invocations --instance-id i-0c30d903851511f06
2026-10-13T13:57:49.827000-03:00 | Success | arn:aws:ssm:sa-east-1:<ACCOUNT_ID>:document/postlab-ec2-remediation-direct-RemediationCommandDocument-oTUx54eMLLIr
CommandId: 2ef2b3a1-073f-4203-93d1-b1c013e3249b
AWS/Events, RuleName=postlab-ec2-remediation-direct-cpu-remediation
Invocations = 1.0
FailedInvocations = 0
InvocationsFailedToBeSentToDlq = 0
The account ID is masked in every captured block in this post. One reading note on those numbers: the same query returned TriggeredRules as 0 for this rule while Invocations was 1, which is the aggregation window of the query and not a contradiction, since the invocation happened and succeeded.
The target is by tag, and that is the point of the template. RunCommandTargets accepts a Key of InstanceIds or tag:<key>, with Values holding 1 to 50 items. Neither field accepts dynamic path syntax, because RunCommandParameters is not among the parameter types that support dynamic path parameters. The identifier of the instance that breached does not arrive here.
What the tag selects is the fifth mark against the direct target, and it only becomes visible when two stacks exist. A tag target set is resolved against the managed nodes of the account and the Region at the moment the command is sent, and nothing in that resolution respects a stack boundary or a VPC boundary, because the command reaches the node through the Systems Manager API and not through the lab network. Both templates deploy a workload node carrying a Workload tag, and this post deploys both. Let them share one tag value and a transition on this stack's alarm sends this stack's document to the other stack's node too, with the IAM condition permitting it, because the condition matches on the same tag that selected the target. That is why every point of use writes !Sub '${WorkloadTagValue}-${AWS::StackName}': the instance tag, the IAM condition in both templates, and the target set here. The stack name is what makes the set a set of one, and it also narrows the Automation execution role in the first template, which would otherwise be allowed to command any node in the account wearing the shared value.
With the tag separated per stack, running both designs side by side is an experiment instead of a collision. Each alarm drives its own node, and the difference between targeting by event-supplied instance ID and targeting by tag stays observable rather than being hidden by a lab that happens to hold one instance.
Four actors, three roles, and what each one is allowed to do
Three roles come out of the template, and one more actor that the template does not create. Confusing two of them is the most frequent mistake in this design.
The instance profile is used by the SSM Agent inside the instance, to register the node and receive commands. Its trust policy names ec2.amazonaws.com and it carries AmazonSSMManagedInstanceCore, plus, in the runbook template only, an inline policy for logs scoped to that stack's log group. That inline policy exists because CloudWatchOutputConfig on the aws:runCommand step sends command output to CloudWatch Logs, and the writer is the agent on the instance using the instance profile credential. AmazonSSMManagedInstanceCore is the baseline for a managed node, and the instance permissions page points at CloudWatchAgentServerPolicy when CloudWatch Logs is in play. The page dedicated to configuring CloudWatch Logs for Run Command is more useful than either, because it publishes the exact statements a custom node policy needs: logs:DescribeLogGroups with no resource scope, and logs:CreateLogGroup, logs:CreateLogStream, logs:DescribeLogStreams and logs:PutLogEvents scoped to a log group prefix. Five actions. A policy carrying three of them deploys without complaint, passes cfn-lint, and then fails at the moment the agent tries to upload, which is worth stating plainly because the whole observability layer of this template hangs off that upload: no output in the log group means no match for the metric filter, and no datapoint for a metric whose alarm treats missing data as not breaching. Silence indistinguishable from success, which is the failure mode this post objects to most. The inline policy here grants all five and narrows the resource from the /aws/ssm/* prefix the page publishes to exactly one known log group. That is the difference between sufficient permission and convenient permission. In the direct target template the agent has nothing to write, so the statement is not there either, which is the least glamorous benefit of a design that cannot collect its own output.
Two details in that inline policy are worth naming. The first is that CloudWatch Logs has two resource types with different ARN formats, and the ARN format in a policy has to match the resource type the API authorizes on. logs:CreateLogGroup and logs:DescribeLogStreams authorize on the log group, whose ARN has no suffix. logs:CreateLogStream and logs:PutLogEvents authorize on the log stream, whose ARN needs either the :* suffix to match every stream in the group or an explicit stream name. That is why the five actions arrive in three statements rather than one.
The second detail looks right and is not. Fn::GetAtt on a log group returns an ARN that already ends in :*, so appending :log-stream:* to it produces an ARN with the wildcard sitting where the log stream name belongs and the rest dangling after it, matching nothing. The log stream scope has to be built out in full, from the partition, region, account and log group name. The access would survive the mistake anyway, because the sibling entry names the ARN that GetAtt returns and that one already matches every stream in the group, and that is exactly why this kind of error gets through a review and a clean deploy without anyone noticing.
The role EventBridge assumes is used by the EventBridge service when it invokes the target. Its trust policy names events.amazonaws.com and conditions on aws:SourceAccount and on aws:SourceArn by rule name prefix. In the runbook template it grants ssm:StartAutomationExecution on the runbook document ARN plus the automation execution namespace, and iam:PassRole on the Automation execution role with iam:PassedToService equal to ssm.amazonaws.com. Passing an execution role to a service requires permission to pass it, which is why that statement is there. I did not find a page stating that requirement specifically for this call, so the statement stays in the template and the sentence stays qualitative.
The resource of the start statement is where the obvious form is the wrong one. A runbook has an automation-definition ARN, the target of the rule uses it, and writing the same ARN into the policy reads like symmetry. The Systems Manager IAM page closes that door: it states that automation definition resources are being deprecated and instructs updating IAM policies to allow ssm:StartAutomationExecution on document and automation-execution resources. So the policy names the document by its exact ARN, which is the form the identity-based policy examples for Automation use, and the execution namespace by wildcard. The price of that is worth naming instead of hiding: automation-execution/* is wider than the single definition ARN it replaced, and there is nothing narrower available, because the execution ID does not exist at the moment the call is authorized. The target ARN on the rule does not move with it. That ARN is how EventBridge identifies the runbook to start, not an IAM resource, and the deprecation note is about IAM policies.
AlarmInvocationRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
Service: events.amazonaws.com
Action: sts:AssumeRole
Condition:
StringEquals:
aws:SourceAccount: !Ref AWS::AccountId
# The rule needs this role ARN, so conditioning on the exact rule
# ARN would be circular. The name prefix is one grade less
# specific and only this stack produces it.
ArnLike:
aws:SourceArn: !Sub 'arn:${AWS::Partition}:events:${AWS::Region}:${AWS::AccountId}:rule/${AWS::StackName}-*'
Policies:
- PolicyName: StartRemediationAutomation
PolicyDocument:
Version: '2012-10-17'
Statement:
# Not automation-definition. The Systems Manager IAM page states
# that automation definition resources are being deprecated and
# instructs allowing StartAutomationExecution on document and
# automation-execution resources instead. The execution namespace
# is a wildcard because the execution ID does not exist yet when
# the call is authorized.
- Sid: StartOnlyThisRunbook
Effect: Allow
Action: ssm:StartAutomationExecution
Resource:
- !Sub 'arn:${AWS::Partition}:ssm:${AWS::Region}:${AWS::AccountId}:document/${RemediationRunbook}'
- !Sub 'arn:${AWS::Partition}:ssm:${AWS::Region}:${AWS::AccountId}:automation-execution/*'
- Sid: PassOnlyTheExecutionRole
Effect: Allow
Action: iam:PassRole
Resource: !GetAtt AutomationExecutionRole.Arn
Condition:
StringEquals:
iam:PassedToService: ssm.amazonaws.com
That trust policy carries a declared compromise. The rule needs the role ARN in RoleArn, and the role's trust policy would ideally condition aws:SourceArn on the rule ARN. Closing that loop with !GetAtt in both directions produces a circular dependency in CloudFormation. The way out is to give the rule an explicit name, !Sub '${AWS::StackName}-cpu-remediation', and to use ArnLike against the prefix only this stack produces. The condition stops naming the exact ARN and starts naming a prefix, one grade less specific in exchange for existing. Naming the rule also means that changing the name requires replacing the resource.
The Automation execution role is used by the Automation service through AutomationAssumeRole, trusts ssm.amazonaws.com with aws:SourceAccount, and grants ssm:SendCommand on the command document ARN plus ssm:SendCommand on instance ARNs conditioned on ssm:resourceTag/Workload. It also grants four read actions with Resource: '*':
- Sid: ReadCommandResults
Effect: Allow
# Granted without a resource scope: these four are read-only and
# read back the result of a command this role itself sent.
Action:
- ssm:GetCommandInvocation
- ssm:ListCommandInvocations
- ssm:ListCommands
- ssm:DescribeInstanceInformation
Resource: '*'
What this role does not carry is any CloudWatch Logs permission, and the absence is deliberate. The step sends an output configuration; the process that writes the output is the agent on the instance, under the instance profile. logs:DescribeLogGroups is required on the instance profile, as above, and on this role it would read as diligence while never being exercised by the execution path, which is the quiet way a least-privilege policy turns into a list of permissions nobody can account for. The two statements are not in conflict: one actor writes and the other one does not.
The fourth actor is the principal that runs the deploy. It needs CAPABILITY_IAM, the ability to create roles and instance profiles, and the ability to pass a role.
The IAM detail that pays for itself
The Run Command documentation records that if you use any global condition keys for the SendCommand action in IAM policies, you must include the aws:ViaAWSService condition key with a boolean value of true.
The practical consequence: ssm:resourceTag/<key> is a service-specific condition key and does not fall under that rule, which is why the tag scoping in both templates works as written. But adding any global condition key to the same statement, something as ordinary as aws:SourceIp or aws:RequestTag, turns SendCommand into a denied call at run time with nothing in the template looking wrong. It is the classic inert condition profile, and here it is inert in the dangerous direction: it denies.
A second detail from the same page, and this one is usable as a test: restricting SendCommand by tag genuinely works, and a command sent to a node that is not tagged, or tagged differently, produces AccessDenied in the execution results. That is a documented mechanism for producing a controlled failure, and the failure scenarios below use it.
A third point, on security. Whoever can send one of the AWS predefined documents to a node has arbitrary execution on it. That is why both templates create their own parameterless command document and grant ssm:SendCommand only on that document's ARN, instead of using AWS-RunShellScript. The policy stops permitting "run anything" and starts permitting "run that script".
Why cfn-lint and validate-template passed a template CloudFormation rejected
The first deploy of the runbook template failed. cfn-lint had been clean on both files in the strict ruleset, and aws cloudformation validate-template had returned exit 0 for both. The stack still rolled back:
$ aws cloudformation deploy --template-file iac/ec2-remediation-automation-runbook.yaml --stack-name postlab-ec2-remediation-runbook
Waiting for changeset to be created..
Waiting for stack create/update to complete
aws: [ERROR]: Failed to create/update the stack.
$ aws cloudformation describe-stacks --stack-name postlab-ec2-remediation-runbook --query 'Stacks[0].StackStatus'
ROLLBACK_COMPLETE
The reason is in the rollback event, and it names the function rather than the resource:
2026-10-13T14:54:27.197000+00:00 ROLLBACK_IN_PROGRESS AWS::CloudFormation::Stack
Template error: every Fn::Equals object requires a list of 2 string parameters.. Rollback requested by user.
No resource failed. The template failed. Fn::Equals compares two strings, and the Rules section was comparing Number parameters against bare YAML integers: RuleCondition: !Equals [!Ref AlarmEvaluationPeriods, 1]. Unquoted, 1 is an integer to the YAML parser, the assertion arrives at the service holding an integer where a string is required, and the whole template is rejected before the first resource is created. The fix is the two quote characters: RuleCondition: !Equals [!Ref AlarmEvaluationPeriods, '1']. Eight numeric literals in the assertions of each template needed them, and both files carry the quoted form, which is why the Rules block earlier in this post reads the way it does. After quoting, both stacks reached CREATE_COMPLETE.
Each validation layer had a reason to let it through, and the three reasons are different.
cfn-lint reads the template as YAML and checks structure, resource types, property names and value domains. !Equals [!Ref AlarmEvaluationPeriods, 1] is a well-formed Fn::Equals with a list of two elements, referencing a parameter that exists. There is nothing malformed to report. The type requirement lives in the function's runtime contract, not in the shape of the node.
validate-template is a service call and it returned exit 0, which is the part worth sitting with. The API validates that the template is syntactically correct and returns the parameters, the description and the capabilities it requires. It does not resolve the Rules section, because rules are evaluated against parameter values at the moment a stack operation is submitted, and validate-template receives no parameter values. So the layer that looks most authoritative is structurally unable to see this class of defect.
That leaves CREATE, which is also the layer that creates resources. The rejection happens before any of them exist, so the cost here was a rollback and not a half-built lab, and that is only because the defect was in a template-level section. A defect in a resource property would have been found after the VPC, the instance and three roles were already up.
The defect survived eight review passes. Every one of them read that block, and the quoting of a numeric literal inside an assertion is not what a reviewer is looking at while checking whether the IAM resource ARN is the deprecated form. It took a CREATE to find it, which is the argument for running the template rather than reviewing it one more time.
Deploying the two stacks
Example values. These are the values the lab actually used, and every block from here on uses them identically:
| Placeholder | Example value |
|---|---|
<REGION> |
sa-east-1 |
<STACK_NAME> |
postlab-ec2-remediation-runbook |
<RUNCOMMAND_STACK_NAME> |
postlab-ec2-remediation-direct |
<ACCOUNT_ID> |
123456789012 inside example ARNs |
WorkloadTagValue prefix |
cpu-remediation-lab |
| Effective workload tag | Workload=cpu-remediation-lab-<STACK_NAME> |
| Load unit | cpu-load-test.service |
Lint and static validation first:
cfn-lint iac/ec2-remediation-automation-runbook.yaml
cfn-lint iac/ec2-remediation-run-command-target.yaml
aws cloudformation validate-template \
--template-body file://iac/ec2-remediation-automation-runbook.yaml \
--region <REGION>
cfn-lint ends with no output and exit code zero when it finds nothing, which is what both files produce, including with the strict ruleset of -c I W E -e --non-zero-exit-code informational. The validate-template exit 0 on record is the one from the previous section, taken before the quoting fix, and I have no run of it against the published form of these files. The verdict that does cover the published form is the CREATE_COMPLETE of both stacks below. Neither lint nor static validation is a promise that the stack will create, as the previous section shows. And validate-template needs a credential, so with none configured it fails with a credential error rather than a template error, which is a different outcome and should not be read as a template problem.
aws cloudformation deploy \
--template-file iac/ec2-remediation-automation-runbook.yaml \
--stack-name <STACK_NAME> \
--parameter-overrides \
InstanceType=t3.micro \
EnableDetailedMonitoring=true \
CpuThresholdPercent=90 \
AlarmPeriodSeconds=60 \
AlarmEvaluationPeriods=3 \
AlarmDatapointsToAlarm=3 \
WorkloadTagValue=cpu-remediation-lab \
--capabilities CAPABILITY_IAM \
--tags Project=posts-aws Source=post-engine Stack=<STACK_NAME> \
--region <REGION>
In the lab, the runbook stack reached CREATE_COMPLETE in 285 seconds on the second attempt, after the quoting fix described above. The direct target stack reached CREATE_COMPLETE in 284 seconds on the first attempt. Most of that time is the instance: the node registered with Systems Manager about 10 seconds after CREATE_COMPLETE, so the stack is usable almost as soon as it reports done.
aws cloudformation describe-stacks \
--stack-name <STACK_NAME> \
--query 'Stacks[0].Outputs' \
--output table \
--region <REGION>
The outputs of the runbook stack are where <INSTANCE_ID>, <RUNBOOK_NAME>, <COMMAND_DOCUMENT_NAME>, <ALARM_NAME>, <RULE_NAME>, <DLQ_URL> and <LOG_GROUP_NAME> come from, plus the effective workload tag, which is worth reading from the output rather than reconstructing by hand, since the parameter is only its prefix. The second stack deploys the same way under <RUNCOMMAND_STACK_NAME>, with the second template file, and its outputs are a shorter list: no runbook name, no execution role and no log group, because it has none of the three. The runbook stack declares 14 outputs and the real stack returned all 14 populated, the direct stack 11.
Teardown is one line per stack:
aws cloudformation delete-stack --stack-name <STACK_NAME> --region <REGION>
aws cloudformation wait stack-delete-complete --stack-name <STACK_NAME> --region <REGION>
aws cloudformation describe-stacks --stack-name <STACK_NAME> --region <REGION> 2>&1 | tail -2
The response to that last command is an error saying the stack does not exist, and that error is the confirmation. Anything else means a leftover. Both deletions ran, and both took the same time:
$ aws cloudformation delete-stack --stack-name postlab-ec2-remediation-runbook
$ aws cloudformation wait stack-delete-complete --stack-name postlab-ec2-remediation-runbook
exit=0, 152s
$ aws cloudformation describe-stacks --stack-name postlab-ec2-remediation-runbook
An error occurred (ValidationError): Stack with id postlab-ec2-remediation-runbook does not exist
$ aws cloudformation delete-stack --stack-name postlab-ec2-remediation-direct
$ aws cloudformation wait stack-delete-complete --stack-name postlab-ec2-remediation-direct
exit=0, 152s
$ aws cloudformation describe-stacks --stack-name postlab-ec2-remediation-direct
An error occurred (ValidationError): Stack with id postlab-ec2-remediation-direct does not exist
A stack-by-stack check is not a sweep, so the account was swept after both deletions: no stack carrying the postlab- prefix, no EC2 instance in a non-terminated state, no EBS volume, no Elastic IP address, no SQS queue and no log group. What remained was exactly the documented residue, two customer managed keys waiting out their window:
54f3bbd5-de7e-4711-8395-a1b37b8a4b52 PendingDeletion 2026-10-20T14:02:20-03:00
8af01f3f-25e6-4ac0-9478-23ad39e1d42f PendingDeletion 2026-10-20T12:14:49-03:00
PendingWindowInDays: 7 produced seven days to the hour, and neither key is a billed item while it waits, because the KMS pricing page exempts a customer managed key scheduled for deletion. They disappear on their own.
What teardown does not undo: the KMS key enters Pending Deletion and stays there for the waiting period, seven days with PendingWindowInDays: 7, and CloudFormation cannot cancel that deletion. Everything else goes. The three stateful resources, the key, the queue and the log group, declare DeletionPolicy: Delete and UpdateReplacePolicy: Delete rather than inheriting them, because the intent of a throwaway lab should be in the file and not in the reader's head. The EBS volume has DeleteOnTermination: true. Two resources here are billed for existing rather than for being used, and the instance is the expensive one by an order of magnitude, so removing the stack is not optional. The other is the KMS key at USD 1 per month prorated hourly, and that charge stops when deletion is scheduled rather than when the waiting period ends. To check the key state:
aws kms describe-key --key-id alias/<STACK_NAME>-lab \
--query 'KeyMetadata.{State:KeyState,Deletion:DeletionDate}' \
--output table --region <REGION>
Verifying the path before generating any load
Each of these confirms a piece of configuration, and none of them confirms delivery.
# the instance is registered as a managed node
aws ssm describe-instance-information \
--filters "Key=InstanceIds,Values=<INSTANCE_ID>" \
--query 'InstanceInformationList[0].{Ping:PingStatus,Agent:AgentVersion,Platform:PlatformName}' \
--output table --region <REGION>
# the command document exists and is the expected type
aws ssm describe-document --name <COMMAND_DOCUMENT_NAME> \
--query 'Document.{Type:DocumentType,Format:DocumentFormat,Status:Status,Target:TargetType}' \
--output table --region <REGION>
# the runbook exists and its parameters match the transformer
aws ssm describe-document --name <RUNBOOK_NAME> \
--query 'Document.{Type:DocumentType,Params:Parameters[].Name}' \
--output json --region <REGION>
# the alarm is configured the way the template declares
aws cloudwatch describe-alarms --alarm-names <ALARM_NAME> \
--query 'MetricAlarms[0].{State:StateValue,Period:Period,Eval:EvaluationPeriods,Dp:DatapointsToAlarm,Missing:TreatMissingData,Threshold:Threshold,Dim:Dimensions}' \
--output json --region <REGION>
# the rule, its pattern and its target
aws events describe-rule --name <RULE_NAME> --output json --region <REGION>
aws events list-targets-by-rule --rule <RULE_NAME> --output json --region <REGION>
# alarm history, empty or carrying only the creation entry
aws cloudwatch describe-alarm-history --alarm-name <ALARM_NAME> \
--history-item-type StateUpdate --max-records 10 \
--output json --region <REGION>
The first one gates every other check, and in the lab it came back about 10 seconds after CREATE_COMPLETE:
$ aws ssm describe-instance-information --filters Key=InstanceIds,Values=i-0e60350eb60ad5cc0
| Id | Ping | Platform | Type | Version |
| i-0e60350eb60ad5cc0 | Online | Amazon Linux | EC2Instance | 3.3.5390.0 |
That agent version is what the Amazon Linux 2023 AMI resolved through the SSM public parameter carried on 2026-10-13. It is not a claim about the newest agent available, and it is not a minimum: it is the version that answered in this run, recorded so the rest of the capture is reproducible against a known starting point.
The remaining expected results, in prose. Any PingStatus other than Online means a problem with the instance profile or with connectivity to the Systems Manager endpoints before it means anything else. The alarm comes back with Period 60, EvaluationPeriods 3, DatapointsToAlarm 3, TreatMissingData set to missing, and the InstanceId dimension carrying the stack's instance. The target comes back with the automation-definition ARN, the invocation role ARN, the input transformer with both paths, and DeadLetterConfig pointing at the queue.
A rule that exists is not proof of delivery. These commands confirm configuration, and nothing more.
The test plan
Seven validations, in order. These are the commands that ran against the runbook stack, with the example values from the table above, and the output they produced follows the block.
One note on the first and last commands, because they look like a contradiction. They start and stop the synthetic load with AWS-RunShellScript, the predefined document this post argues against granting to the remediation path. That is deliberate and it is a different actor: starting a test load is an operator action taken with an operator's credential, while the remediation path runs unattended with a role that exists only to send one document. The argument is about what the automation is allowed to do, not about what a person debugging a lab is allowed to type.
# confirm the dead letter queue is empty before starting
aws sqs get-queue-attributes --queue-url <DLQ_URL> \
--attribute-names ApproximateNumberOfMessages \
--output table --region <REGION>
# start the synthetic load on the lab instance
aws ssm send-command \
--document-name AWS-RunShellScript \
--instance-ids <INSTANCE_ID> \
--parameters 'commands=["systemctl start cpu-load-test.service","systemctl is-active cpu-load-test.service"]' \
--comment "start synthetic cpu load" \
--output json --region <REGION>
# read the metric during the load window
aws cloudwatch get-metric-statistics \
--namespace AWS/EC2 --metric-name CPUUtilization \
--dimensions Name=InstanceId,Value=<INSTANCE_ID> \
--statistics Average --period 60 \
--start-time <START_ISO8601> --end-time <END_ISO8601> \
--output table --region <REGION>
# the state transition, with its timestamp
aws cloudwatch describe-alarm-history --alarm-name <ALARM_NAME> \
--history-item-type StateUpdate --max-records 10 \
--output json --region <REGION>
# the Automation execution and its input parameters
aws ssm describe-automation-executions \
--filters Key=DocumentNamePrefix,Values=<RUNBOOK_NAME> \
--output json --region <REGION>
aws ssm get-automation-execution --automation-execution-id <EXECUTION_ID> \
--query 'AutomationExecution.{Status:AutomationExecutionStatus,Params:Parameters,Steps:StepExecutions[].{Name:StepName,Status:StepStatus,Msg:FailureMessage,Out:Outputs}}' \
--output json --region <REGION>
# the Run Command invocation, on the right instance
aws ssm list-command-invocations --command-id <COMMAND_ID> --details \
--query 'CommandInvocations[].{Instance:InstanceId,Status:Status,Code:ResponseCode,Out:CommandPlugins[0].Output}' \
--output json --region <REGION>
aws ssm get-command-invocation --command-id <COMMAND_ID> --instance-id <INSTANCE_ID> \
--query '{Status:Status,Code:ResponseCode,Stdout:StandardOutputContent,Stderr:StandardErrorContent}' \
--output json --region <REGION>
# the command output in CloudWatch Logs
aws logs filter-log-events --log-group-name <LOG_GROUP_NAME> \
--filter-pattern '"REMEDIATION_"' --limit 20 \
--output json --region <REGION>
# recovery: the alarm back in OK
aws cloudwatch describe-alarms --alarm-names <ALARM_NAME> \
--query 'MetricAlarms[0].{State:StateValue,Reason:StateReason,Updated:StateUpdatedTimestamp}' \
--output json --region <REGION>
# confirm no invocation failure happened
aws cloudwatch get-metric-statistics \
--namespace AWS/Events --metric-name FailedInvocations \
--dimensions Name=RuleName,Value=<RULE_NAME> \
--statistics Sum --period 300 \
--start-time <START_ISO8601> --end-time <END_ISO8601> \
--output table --region <REGION>
# stop the load, in case it is still running
aws ssm send-command \
--document-name AWS-RunShellScript \
--instance-ids <INSTANCE_ID> \
--parameters 'commands=["systemctl stop cpu-load-test.service","systemctl is-active cpu-load-test.service || true"]' \
--comment "stop synthetic cpu load" \
--output json --region <REGION>
What the run produced
The load started at 12:02:01 and the alarm transitioned 245 seconds later, which the section on alarm delay covers minute by minute. What follows the transition is the part that is not arithmetic:
| Timestamp | Event | Where it was read |
|---|---|---|
| 12:06:06.831 | alarm OK to ALARM
|
describe-alarm-history |
| 12:06:07.105 | Automation execution started | describe-automation-executions |
| 12:06:08 | remediation script started | command output |
| 12:06:17.804 | Automation execution Success
|
get-automation-execution |
| 12:08:06.894 | alarm ALARM to OK
|
describe-alarm-history |
274 milliseconds from the alarm transition to the execution start, and 10.7 seconds for the execution itself. The CPUUtilization average was 94.9901% in the 12:06 minute, the one the remediation interrupted, and 0.2000% in 12:07.
The rule did its job once, and the dead letter queue stayed empty:
AWS/Events, RuleName=postlab-ec2-remediation-runbook-cpu-remediation
TriggeredRules = 1.0
Invocations = 1.0
FailedInvocations = 0
DLQ: ApproximateNumberOfMessages = 0, ApproximateNumberOfMessagesNotVisible = 0
The command output reached the log group, in its own stream named after the command, the instance and the document step:
$ aws logs describe-log-streams --log-group-name /aws/ssm/postlab-ec2-remediation-runbook-remediation
868279b1-2cc6-4d63-99d5-bc2277b60e50/i-0e60350eb60ad5cc0/stopTestLoad/stdout
$ aws logs filter-log-events --log-group-name /aws/ssm/postlab-ec2-remediation-runbook-remediation
REMEDIATION_START unit=cpu-load-test.service instance=i-0e60350eb60ad5cc0 at=2026-10-13T15:06:08Z
REMEDIATION_ACTION stopped=cpu-load-test.service
REMEDIATION_OK cpu=0
REMEDIATION_OK cpu=0 is the script measuring the condition after acting, so the four outcomes in the paragraph below did not diverge in this run: the execution succeeded, the script exited zero, and the condition was gone. The execution parameters and the step inputs are in the section on where the instance ID lives, because that is the claim they settle.
The load generator is a systemd unit, and the start command reported the unit active on a node with nproc equal to 2, which is the configuration the one-loop-per-vCPU argument depends on.
How long to wait before concluding that something did not work. With detailed monitoring, CPUUtilization is published in 1-minute periods, and the transition requires DatapointsToAlarm breaching data points inside EvaluationPeriods periods, which here is 3 of 3. So the floor is three minutes of sustained load plus the publishing delay, and the operational guidance is to wait five to seven minutes before concluding anything. That five to seven is the arithmetic plus observation slack, not a measured number. Concluding early is the most common mistake for anyone testing this for the first time.
One more thing that saves time: the Run Command API follows an eventual consistency model because of the distributed nature of the system behind it, so the result of a call may not be immediately visible to following calls. A list-command-invocations that comes back empty right after send-command does not mean the command does not exist.
Four things that get confused, and this is the most useful paragraph in the section. An Automation execution succeeding means the flow ran and no step aborted. The script succeeding means ResponseCode is zero. The remediation being operationally effective means the condition stopped existing, which is what REMEDIATION_OK versus REMEDIATION_FAILED measures. The application recovering means the application is serving again, and nothing in this design measures that. All four can diverge, and the case that matters is an automation reporting Success while CPU is still above 90 percent. With aws:runCommand that does not happen silently, because the script's non-zero exit code fails the step. With aws:executeAwsApi calling SendCommand it does happen: the call returns the CommandId and the automation moves on without waiting for the command result.
Failure scenarios and what each one looks like
| Scenario | How to provoke it | Where the evidence appears | Mitigation in the design |
|---|---|---|---|
| Node not registered as a managed node | remove the instance profile from the instance |
describe-instance-information stops listing the node; the aws:runCommand step fails with a FailureMessage
|
the script failure alarm does not cover this; the automation execution failure rule does |
| Insufficient IAM permission on the execution role | remove the Workload tag from the instance so the ssm:resourceTag/Workload condition stops matching |
AccessDenied in the execution results |
this is the tag scoping working, and it doubles as a positive test of the scope |
| Insufficient permission on the EventBridge invocation role | temporarily remove the ssm:StartAutomationExecution statement |
the message goes straight to the dead letter queue with no retry, carrying ERROR_CODE of NO_PERMISSIONS or FAILED_TO_ASSUME_ROLE
|
the queue, plus the queue depth alarm |
| Instance identifier never reaches the runbook | point the alarm at a metric without the InstanceId dimension |
a required parameter is missing, the rule is created and does not run | none; this is the structural limit of the design |
| An invalid instance identifier reaches the runbook | point the alarm at a metric whose InstanceId dimension carries something that is not an instance identifier |
the execution is rejected on the parameter rather than on the command | the allowedPattern on the runbook's InstanceId parameter |
| Script returns a non-zero exit code | keep the condition present through another means |
ResponseCode other than zero, REMEDIATION_FAILED in the log, the step in Failed because of onFailure: Abort
|
in the runbook template, the metric filter, the script failure alarm and the execution failure rule; in the direct target template, only the command invocation status |
| CPU stays above 90 percent after remediation | same as above |
REMEDIATION_FAILED cpu_still_high=<value> in the command output |
in the runbook template, this is the exact case the metric filter counts; in the direct target template the marker exists only in the command output retrieved through the Run Command APIs |
| Repeated transitions firing multiple executions | cycle the load | several automation executions, one per OK to ALARM transition | the idempotent script, and the remediation loop discussion below |
| Duplicate event delivery | not provokable deterministically | two executions with identical parameters | the idempotent script, and MaximumEventAgeInSeconds bounding the window |
| Command never reaches the agent | stop the SSM Agent, then fire the alarm | command status Delivery Timed Out after the step's TimeoutSeconds, which accepts 30 to 2592000 seconds and is a delivery timeout, not an execution timeout |
the step fails, the automation ends in Failed, and the execution failure rule matches Failed
|
| Script runs longer than it should | lower timeoutSeconds inside the command document below the script duration |
the command invocation reports the timed out plugin, not a delivery timeout | the step fails, the automation ends in Failed, and the execution failure rule matches Failed
|
| The failure notification itself is not published | remove the AllowFailureRuleToPublish statement from the topic policy |
FailedInvocations on the execution failure rule's dimension, then the message in the dead letter queue |
the retry policy and the dead letter queue on that rule's target, plus the alarm on its own FailedInvocations
|
| Failure to write to the dead letter queue itself | this is the encryption key scenario above |
InvocationsFailedToBeSentToDlq non-zero |
the dead letter write failure alarm |
The last one matters most, for one reason: it is the only failure that erases the record of the previous failure.
Automation runbook or direct Run Command
On targeting, the runbook takes the identifier from the event through the input transformer, and the direct target takes a static InstanceIds list or a tag:<key> match with no dynamic path syntax available, resolved across the account and the Region rather than inside the stack. That is the whole decision for the requirement stated at the top of this post.
On extra resources, the runbook costs one Automation document and one execution role, and the direct target costs nothing extra. On IAM, the runbook needs ssm:StartAutomationExecution plus a role to pass, and the direct target needs ssm:SendCommand. On error handling, the runbook has onFailure, finallyStep and chainable steps, and the direct target has the command status and nothing else. On execution visibility, the runbook emits execution and step events carrying ExecutionId, StepName, Action and Status, and returns CommandId, Status, ResponseCode and Output as step outputs; the direct target emits command and command invocation events, the second of which carries instance-id.
On cost, Automation is billed at USD 0.002 per step executed and USD 0.00003 per second of script executed, and Run Command carries no additional charge. On scale, Automation has rate controls with a concurrency value and an error threshold, and an opt-in adaptive concurrency mode scales the quota from 100 concurrent automations up to 500; Run Command has a default MaxConcurrency of 50 and accepts up to 50 instance IDs per call. On cancellation, a CloudWatch alarm can be attached to an automation so that an ALARM state cancels the automation and runs the defined onCancel steps; there is no equivalent for a direct target.
Automation earns its place when the target has to come from the event, when there is more than one step, when a step's result feeds the next step, or when per-step history is wanted. A direct Run Command target is enough for a single-step procedure with a known or tag-defined target and a script that is idempotent and harmless on a healthy node.
Which answers a question worth being precise about. The common design is appropriate because it meets the stated requirements, not because it is universally better. Those are different claims. Replace the requirement "remediation runs on the instance that breached" with "remediation runs on the set", and the direct target becomes the better choice by the same criteria.
Credit, and two places where this design diverges from the official walkthrough
The path through the input transformer is documented in a walkthrough on the AWS Cloud Operations blog, Use Amazon EventBridge rules to run AWS Systems Manager Automation in response to CloudWatch alarms, published in August 2021. That post is where the JSON path in this design comes from, and the design here diverges from it in two places, with the reason named for each.
The example runbook there uses aws:executeAwsApi calling send_command. That action fires and moves on: the step receives the CommandId and finishes. This design uses aws:runCommand, which is the purpose-built action and returns the command's Status, ResponseCode and Output. The difference is that with the first one the automation can report success for a script that failed.
The walkthrough also instructs attaching AmazonSSMFullAccess and, in the same breath, recommends least privilege as good practice. This design ships the scoped policies that recommendation asks for. The permissions in those policies were assembled from the API actions involved, not copied from a page that enumerates them, because no such page exists for an EventBridge target of Systems Manager Automation.
Diverging from an official example is not correcting AWS. It is the difference between a walkthrough that has to fit on a screen and a template headed for production.
High CPU is not necessarily a failure
Ninety percent CPU describes a condition, not a defect. The possible causes include legitimate load growth, increased concurrency, work that is compute-intensive by nature, inefficient code, resource contention and insufficient capacity. Exactly one of those is solved by terminating a process.
What the metric measures matters here. CPUUtilization is the percentage of allocated physical CPU time the instance uses, covering user code and EC2 code, and at a high level it is the sum of guest and hypervisor utilization. Operating system tools can report a different percentage, for reasons including legacy device simulation, non-legacy device configuration, interrupt-heavy workloads, live migration and live update. A remediation triggered by that number is reacting to a measurement taken from outside the instance, which is not the same number the operator sees inside it.
Five possible responses, and where each one fits.
Automated remediation, which is this post, fits when there is a specific, known, idempotent corrective procedure for an identified cause. A process that leaks, a queue that jammed, a job that never finished.
Auto Scaling fits when the 90 percent is legitimate demand. The policy tracks the metric and defines the action while the alarm is in ALARM, over an aggregation of the metrics from all instances in the group. It is the right answer for capacity and the wrong answer for a stuck process.
Application-level recovery, meaning an application health check, a supervisor restarting the process, a circuit breaker, fits when the condition belongs to the application and the application knows how to recover.
EC2 automatic instance recovery addresses an underlying hardware or software failure that makes the system status check fail, is monitored through StatusCheckFailed_System, and does not operate when only the instance status check fails. High CPU triggers none of it. Worth knowing what recovery preserves and loses, since the question comes up: a recovered instance keeps its instance ID, its public, private and Elastic IP addresses, its metadata, its placement group, its attached EBS volumes and its availability zone, and loses whatever was in volatile memory.
Human incident response fits when the cause is unknown. This is where creating an OpsItem as the alarm action is worth more than executing a script.
Two sentences on why remediating from a single metric introduces risk. A remediation that terminates a process whenever CPU passes 90 percent turns a capacity problem into a recurring incident, because the condition returns and the remediation returns, and the chart of automation executions becomes the only place where the real problem is visible. Correlating CPU with application health, latency and error rate changes the decision, and the native mechanism for that is the composite alarm, which goes into ALARM only when the conditions of its rule expression are met and exists precisely to reduce alarm noise.
The honest version of the design, then: a CPU alarm alone drives remediation. A composite alarm requiring high CPU and high latency and an error rate above normal drives remediation with far less chance of reacting to commercial success. The cost is that a composite alarm cannot perform EC2 or Auto Scaling actions, and that it needs application metrics which do not arrive for free. The composite version is better and it is not cheaper.
Production readiness
On security: least privilege with named actions and tag scoping, four separate actors, a purpose-built command document instead of AWS-RunShellScript, no document parameters and therefore no injection surface, no inbound port and no long-lived credential with the SSM Agent opening every connection, and encryption at rest on the queue, the topic and the root volume. The CloudTrail record of this path is StartAutomationExecution, SendCommand and AssumeRole, plus StopAutomationExecution when an attached alarm cancels an automation. Neither template creates a trail, and that is a decision rather than an omission: those four are management events, so they already appear in the account's event history without one, and a lab-scoped trail would add an S3 bucket to a teardown this post promises in a single command. And the aws:ViaAWSService trap described above, which is the one that bites silently.
On reliability: an explicit RetryPolicy and a dead letter queue carrying the diagnostic attributes on both rules rather than only on the remediation one, idempotency from systemctl stop, concurrent execution bounded by rate controls, with an opt-in adaptive concurrency mode that scales the quota from 100 to 500, and both timeouts declared rather than defaulted, the delivery one on the step and the duration one inside the command document. There is no automatic rollback: remediation does not undo. Escalation is the execution failure rule notifying the topic, and that rule is instrumented like the path it reports on, with its own FailedInvocations alarm, because an escalation channel that fails silently is worse than no escalation channel. The next step from there would be an OpsItem or Incident Manager, both of which are supported EventBridge targets.
On observability, for the runbook path: the CPU metric, alarm history retained for 30 days, the rule metrics in AWS/Events with the caveat that several are sent only when non-zero, the Automation execution history, the command output in CloudWatch Logs, and the metric filter counting script failures. The direct target path gives up the last three, which is the single largest difference between the two in production and the one that would be felt during an incident rather than during a review. What is opt-in and has to be said out loud: EC2 detailed monitoring, CloudWatchOutputConfig on the step, and the alarms themselves. None of that arrives enabled.
On governance: change management through CloudFormation. For remediation approval, Automation supports a manual approval step, which is the path when the procedure is destructive. Execution limits come from the rate controls. For a maintenance window, Systems Manager Change Calendar emits Calendar State Change with a state of OPEN or CLOSED, which is the native mechanism for suppressing remediation during a window. Auditability comes from CloudTrail and the events. Ownership is the template.
On cost, the structure rather than a total. What dominates the bill in this design is the EC2 instance, by the hour, and nothing else comes close. Run Command carries no additional charge. Automation costs USD 0.002 per step executed and USD 0.00003 per second of script executed, both from the announcement of August 14, 2025, which is also where the end of the free tier is recorded. EC2 detailed monitoring, CloudWatch alarms, log retention and messages in SQS and SNS are billed items whose unit prices are not quoted here because I did not confirm them on an official pricing page. The KMS key is the exception and it is a floor rather than consumption: USD 1 per month prorated hourly while the key exists, and nothing while it waits out a scheduled deletion. The number of CloudWatch alarms is the one that changes decisions: this design is per instance, so the alarm count grows with the fleet. EventBridge participates in the path, and this post makes no claim about what that costs.
What was verified, and by which tool
Four layers ran on this material, and they do not overlap as much as they look.
cfn-lint 1.46.0 on both templates, in the default ruleset and in -c I W E -e --non-zero-exit-code informational: empty output and exit 0 in all four runs. That covers template structure, resource types, property names and value domains, and the previous section is the record of what it does not cover.
A shell syntax check on the scripts that live inside the templates as YAML string lists. cfn-lint validates the YAML and the resource properties, and does not parse those strings as shell, so a syntax error there would appear for the first time on a node, with the command already dispatched. I reconstructed the remediation script and the load script from the lists and ran sh -n on both: exit 0 on both.
A unit test of the arithmetic the remediation script uses to decide between REMEDIATION_OK and REMEDIATION_FAILED. The script reads /proc/stat twice and computes busy time over total time, so I fed the same arithmetic controlled samples instead of a live node: 50%, 100%, 0%, 95%, 89%, and the case where no ticks elapsed between the two reads, which is the division by zero the script has to survive. Six cases, six correct, and the threshold behaves as declared at the boundary, with 89% producing REMEDIATION_OK and exit 0 while 90% produces REMEDIATION_FAILED and exit 1.
Both stacks deployed, driven with real load and destroyed in sa-east-1 on 2026-10-13, with the output published above.
What was not exercised, stated so nobody has to guess. The failure path is documented and not observed: the agent unavailable, insufficient IAM permission on either role, an invalid target, a script exiting non-zero, CPU that stays high after the remediation, repeated transitions, duplicate delivery and an Automation delivery timeout were all left unprovoked. Because nothing failed, the dead letter write was never attempted, so its reachability is untested rather than confirmed. The version-agnostic selector in the target ARN was accepted at create time and nothing in the capture describes how it resolved. And the two claims about scoping the key policy by source stayed open, because breaking them on purpose is the only way to read the result.
Where these services are heading
Five movements, each with a source, because a design described against last year's documentation teaches the wrong thing.
The IAM resource type for starting an automation is on its way out, and this is the only movement here that changes a line in the templates. The Systems Manager IAM page, which is also where the automation-definition ARN form is documented, carries a note that uses the documentation's own word: automation definition resources are being deprecated, and IAM policies should be updated to allow ssm:StartAutomationExecution or ssm:StartChangeRequestExecution on document and automation-execution resources. The identity-based policy examples that note points at name the document ARN and say nothing about automation-definition. The failure mode on the day the deprecation lands is AccessDenied on the start call, which in this design surfaces as FailedInvocations on the rule and a dead letter message carrying a permission error code, not as a visible break. The invocation role in the first template is written in the new form for that reason.
Managed node permissions moved. The Systems Manager documentation presents Default Host Management Configuration as the recommended configuration for EC2 instance permissions and presents the instance profile as the alternative. With Default Host Management Configuration turned on, every instance in the region and account using IMDSv2 with SSM Agent 3.2.582.0 or later becomes a managed node automatically, with no instance profile. IMDSv1 is not supported. The documentation does not call the instance profile legacy or deprecated, and neither does this post. These templates use an instance profile anyway, for a reason worth stating: Default Host Management Configuration is an account and region setting that applies to all instances, and a lab template that turned it on would be changing configuration outside its own scope.
The agent's messaging endpoint moved. Starting with SSM Agent version 3.3.40.0, Systems Manager uses the ssmmessages:* endpoint whenever it is available, instead of ec2messages:*, and that holds even when the instance profile is configured to allow both. Practical consequence, in two parts. In Regions launched in 2024 and later only ssmmessages:* is supported, so an endpoint or permission list carrying ec2messages describes a Region that cannot use it. In Regions launched before 2024 the documentation states that ec2messages:* permissions can safely be removed now, and that an agent holding both sees no calls to ec2messages:* because it connects to ssmmessages anyway. The prose earlier in this post lists all three interface endpoints because that is the list the connectivity page presents, and the templates grant permissions through AmazonSSMManagedInstanceCore rather than by naming endpoints, so neither file has to choose.
The Automation free tier ended, with a date. The announcement of August 14, 2025 records that the existing Automation free tier of 100,000 steps and 5,000 seconds of script duration per month stopped being available to new customers and ended on December 31, 2025 for existing ones. The same announcement introduced automatic retry of throttled API calls in high-concurrency scenarios, which is a direct reliability improvement for this design.
Adaptive concurrency exists as an opt-in. The documentation presents it as a mode you can opt in to, and states that it automatically scales the automation quota from 100 concurrently running automations up to 500. It is not on by default, and the documentation does not present it as a recommendation. It matters here because per-instance remediation multiplies executions with fleet size, and the concurrency quota is the ceiling that shows up first.
One more note, and it needs care. The EventBridge targets documentation now carries a note saying the topic describes targets for Custom Event Bus - Classic, and that on the Custom Event Bus a target belongs to a subscriber rather than to a rule. The chapter on the Custom Event Bus is where the relationship between the two products is stated: they share the events: IAM namespace and the events.amazonaws.com service principal, both remain available, and there is a documented path for moving existing rules to subscribers. This post uses the default event bus, which is where events from AWS services necessarily arrive, and the note on the targets page concerns the Custom Event Bus. Neither page calls AWS::Events::Rule legacy or deprecated, and neither does this post.
Results matrix
Both templates pass cfn-lint. The aws cloudformation validate-template exit 0 belongs to the bytes before the quoting fix, so the service verdict on the published form is the one that follows: both stacks were created in sa-east-1 on 2026-10-13, driven with a synthetic CPU load, and deleted in the same run. Nine rows carry captured output. The last four stay open, and they are three separate experiments. The dead letter write was never attempted, because nothing in the path failed. The version selector in the target ARN is not named anywhere in the capture, which records only a stack that reached CREATE_COMPLETE carrying it. And the final two are one experiment taken from both sides: adding a source condition to the key policy and redeploying to watch what breaks. One claim from the research also stays open without a row of its own, for the same reason the dead letter row does: whether a queue left on the default SQS-managed encryption is reachable by EventBridge at all.
| Validation | Expected result | Actual result | Evidence, or why there is none |
|---|---|---|---|
| Synthetic load generated |
cpu-load-test.service active, one busy loop per vCPU |
unit reported active on a node with nproc equal to 2 |
send-command running systemctl start and systemctl is-active
|
| CPU metric above the threshold |
Average above 90 in consecutive 1-minute periods |
0.0030% and 0.2164% before the load, then 15.0042%, 100.0%, 99.9967% and 100.0% |
get-metric-statistics, Average, 60-second periods, 12:00 to 12:05 |
| Alarm transitioned to ALARM |
OK to ALARM after at least three breaching datapoints |
OK to ALARM at 12:06:06.831, 245 seconds after the load started |
describe-alarm-history |
| Event matched the rule |
TriggeredRules non-zero, FailedInvocations absent |
TriggeredRules = 1.0, Invocations = 1.0, FailedInvocations = 0
|
AWS/Events with RuleName=postlab-ec2-remediation-runbook-cpu-remediation
|
| Automation execution started | one execution with InstanceId equal to the breaching instance |
execution 70b6fea9-cec2-41c8-aaac-60cabdfe3921, started 274 ms after the transition, Success 10.7 seconds later |
get-automation-execution, Parameters carrying InstanceId as i-0e60350eb60ad5cc0
|
| Run Command executed on the right instance |
ResponseCode 0, InstanceId matching |
step runRemediation returned Success with InstanceIds as ["i-0e60350eb60ad5cc0"], and a non-zero exit would have failed the step |
step Inputs and Outputs, plus the log stream 868279b1-2cc6-4d63-99d5-bc2277b60e50/i-0e60350eb60ad5cc0/stopTestLoad/stdout
|
| Remediation resolved the condition |
REMEDIATION_OK in the command output, alarm back to OK
|
REMEDIATION_OK cpu=0, and the alarm back to OK at 12:08:06.894 |
filter-log-events on the command log group, describe-alarm-history
|
Run Command target accepted with no Input field |
the transition produces a command invocation on the tagged node | command 2ef2b3a1-073f-4203-93d1-b1c013e3249b dispatched to i-0c30d903851511f06 and reported Success
|
list-command-invocations, plus Invocations = 1.0 and FailedInvocations = 0 on the direct stack's rule |
| Command output in the direct target stack | no log group receives it, and the output is available only through the Run Command APIs |
CloudWatchOutputEnabled: False with an empty log group name, and no /aws/ssm log group in the Region |
get-command-invocation, describe-log-groups --log-group-name-prefix /aws/ssm returning nothing |
| Dead letter queue reachable by EventBridge | a failed invocation becomes a message in the queue | Not executed | not provoked: no invocation failed, so EventBridge never attempted the write, and an empty queue is consistent with reachability without being evidence of it |
$DEFAULT accepted in the automation-definition target ARN |
the service resolves the version-agnostic selector at invocation time | Not executed | not provoked: the capture records the stack carrying that ARN reaching CREATE_COMPLETE, which is acceptance at create time, and names nothing about how the selector resolved |
| Source-scoped KMS key policy and the dead letter write | adding aws:SourceAccount to the EventBridge key statement breaks the write |
Not executed | not provoked: reading the result means shipping the condition and then failing an invocation on purpose, which is a second run with a stack built to lose a record |
| Source-scoped KMS key policy and the alarm notifications | the five operational alarms publish with aws:SourceArn scoped to the stack's alarm name prefix |
Not executed | not provoked: same experiment from the other side, since it needs the scoped policy deployed and an operational alarm transition to observe |
The four open rows stay in the table rather than leaving the post. Two of them cost a deliberate break to answer, one needs a failure the happy path does not produce, and one needs a reference page I did not find. The nine above them were inferences from documentation before this run, and one of them, the output configuration of the direct target, is the kind of claim I would not have trusted without reading False in a response.
Where this design stops serving
This design remediates one node because the identifier comes from the alarm's own dimensions, so the next problem is the lifecycle of one alarm per instance. At a few dozen nodes the question stops being how to remediate and becomes how to create and retire alarms as instances come and go, which points at alarm creation driven by instance lifecycle events, or at per-service application metrics instead of per-node CPU. From there the conversation moves to composite alarms correlating CPU with latency and error rate, and then to the point where the honest answer stops being remediation and becomes capacity. And if the right answer turns out to be replacing the instance rather than repairing it, the subject stops being Systems Manager and becomes Auto Scaling health checks and immutable instances.
The concrete gain from the two templates is narrow and worth having. One command stands up a reproducible CPU remediation path with scoped roles, a dead letter queue that EventBridge can actually write to, and five alarms covering the places where this kind of path fails silently. One command takes it down, minus a KMS key that waits seven days. And the second template is why the claim about targeting is measured in this post rather than asserted.
References
- Amazon CloudWatch and Amazon EventBridge
- Use Amazon EventBridge rules to run AWS Systems Manager Automation in response to CloudWatch alarms, AWS Cloud Operations blog
- Amazon EventBridge targets
- Amazon EventBridge event bus (Custom), the source of the shared namespace, the shared service principal and both products remaining available
- EventBridge RunCommandTarget API reference
- Transforming Amazon EventBridge target input, including the quoting rules for string variables, the behavior of a JSON path that does not resolve, and the note that SSM targets do not support Input and InputPath
- EventBridge PutRule API reference, including the 64 character rule name limit
- Running automations with Amazon EventBridge
- Tutorial: using EventBridge input transformers with Automation
- AWS::Events::Rule Target
- AWS::Events::Rule RetryPolicy
- Event retry policy and using dead letter queues in EventBridge
- Monitoring Amazon EventBridge with CloudWatch metrics
- AWS::CloudWatch::Alarm
- AWS::CloudWatch::Alarm WarmUpConfiguration, the source of the statement that an alarm stays in INSUFFICIENT_DATA and performs no actions during the warm-up period
- Using Amazon CloudWatch alarms
- CloudWatch PutMetricAlarm API reference, including the recommendation to omit Unit
- Alarm actions supported by CloudWatch
- Configuring a CloudWatch alarm to invoke a Lambda function
- Creating metrics from log events using filters
- Using identity-based policies for CloudWatch Logs, including the log-group and log-stream ARN formats and the log-stream-level policy examples
- AWS::Logs::MetricFilter
- List the available CloudWatch metrics for your Amazon EC2 instances
- Manage detailed monitoring for your Amazon EC2 instances
- Recover your Amazon EC2 instance
- Dynamic scaling for Amazon EC2 Auto Scaling
- Step and simple scaling policies
- Health checks for instances in an Auto Scaling group
- Replace unhealthy instances in an Auto Scaling group
- Gracefully handle Amazon EC2 instance termination
- aws:runCommand, run a command on a managed node
- AWS::SSM::Document
- Configure instance permissions required for Systems Manager
- Improve the security of EC2 instances by using VPC endpoints for Systems Manager
- Reference: ec2messages, ssmmessages, and other API operations, including the 3.3.40.0 endpoint connection precedence and the Region split
- Amazon Machine Images with SSM Agent preinstalled
- Finding public parameters for Amazon Linux AMIs
- Setting up Automation
- How AWS Systems Manager works with IAM, the source of the automation-definition ARN form and of the note that automation definition resources are being deprecated
- Setting up identity based policies examples for Automation, the source of the document ARN form that replaces it
- Using Automation with AWS CloudFormation and service roles
- Running automations at scale, rate controls and adaptive concurrency
- Configuring Automations to monitor CloudWatch alarms, the source of the cancellation behavior, the onCancel steps and the ssm:StopAutomationExecution permission it needs
- AWS Systems Manager Run Command
- SendCommand API reference, including the 100 character comment limit
- Setting up Run Command, including the aws:ViaAWSService requirement
- Configuring Amazon CloudWatch Logs for Run Command, the source of the five log actions a custom node policy needs
- Command document plugin reference, including interpolationType
- SSM document schemas and features, including allowedPattern and the parameter security best practices
- Example Systems Manager events for Amazon EventBridge
- Reference: Amazon EventBridge event types for Systems Manager, including the closed list of Automation status values
- Cross-service confused deputy prevention in EventBridge, the source of the target role trust policy conditions
- Notifying users of alarm changes, including preventing confused deputy issues on the topic policy
- AWS CloudFormation template Rules section
- AWS Systems Manager Automation enhances runbook experience, announcement of August 14, 2025, with pricing and free tier change
- AWSSupport-TroubleshootCloudWatchAlarm runbook
- AWS::SQS::Queue, including SqsManagedSseEnabled default behavior
- Available CloudWatch metrics for Amazon SQS
- AWS::KMS::Key, including PendingWindowInDays
- AWS KMS pricing, the source of the USD 1 per month prorated hourly charge per key and of the exemption for a key scheduled for deletion
- Managing Amazon SNS encryption keys, both the instruction to add the source condition keys to the KMS key policy and the exception for EventBridge-to-encrypted topics
- Well-Architected Framework, Operational Excellence pillar
- Well-Architected Framework, Security pillar
- Well-Architected Framework, Reliability pillar
- Well-Architected Framework, Performance Efficiency pillar
- Well-Architected Framework, Cost Optimization pillar
- Well-Architected Framework, Sustainability pillar
Top comments (0)