DEV Community: Carlos Filho

Scheduling weekly SNS notifications with Amazon EventBridge

Carlos Filho — Mon, 19 Oct 2026 21:07:39 +0000

A manual weekly SNS campaign was replaced by a scheduled EventBridge trigger in two CloudFormation templates: an Events rule in UTC and a Scheduler in local time.

Every Friday around 11 PM, someone on the marketing team of a retail client opened the Amazon SNS console, pasted the Saturday campaign copy into the message field and clicked publish. The topic carried a few million confirmed subscriptions, in a mix of email, SMS, mobile push and one SQS queue feeding an internal consumer. The procedure lived in a shared document, with the topic ARN in bold and an all-caps warning to confirm the region before publishing.

What brought me in was not a push for automation. It was an incident. On two consecutive Saturdays the campaign went out twice. The first time, the person published, the page stopped responding, and she published again. The second time, two people each assumed the other was on vacation. Between those two episodes there was a Saturday when the campaign did not go out at all, and nobody noticed until Monday, when the conversion report came back at zero.

What I found when I opened the account: no alarms, and no record of who published what beyond raw CloudTrail. The delivery mechanism worked fine. The operational knowledge sat with two people, and the task runs 52 times a year, always at the same hour, finishing in under a second.

What had to change, and what could not

I needed a reliable weekly trigger, with no server to maintain, that leaves a trace of every firing. The delivery mechanism was not up for discussion: the SNS topic, the confirmed subscriptions and the endpoints stay exactly as they are. Any design that would require re-enrolling millions of subscribers was out before we started.

Requirements for the weekly trigger

The firing happens with no human in the loop, on Saturday, at a fixed local time in Sao Paulo. The hour itself moved: the manual ritual published late on Friday night so the message would be waiting when subscribers woke up, and the marketing team agreed to 09:00 on Saturday instead, which is a better hour for the campaign and a far easier one to reason about in a cron expression.
No new compute resource to maintain, patch or monitor.
The existing SNS topic stays as the delivery point, with its subscriptions untouched.
The topic supports subscriptions in the order of millions, and the scheduling layer cannot be the bottleneck.
A failed firing produces recoverable evidence, with payload and error code, instead of disappearing.
No principal outside the account can publish to the campaign topic.
The scheduling layer carries no fixed hourly cost, so the time it sits idle between firings is not billed.
Every firing is observable through a metric and an alarm, without depending on someone watching.

Four designs for the same weekly trigger

Cron on a dedicated EC2 instance

This is the design anyone with an infrastructure background reaches for first, and it does work. The team already knows how to write a script and how to edit a crontab. The script can do anything before publishing, including querying a database inside the VPC to assemble the message body. There is no learning curve, and there is full control over the payload.

What breaks is the host. A single instance sits on the critical path of a weekly campaign. If that instance is stopped, degraded, halfway through a reboot, or replaced by an Auto Scaling event during the scheduled minute, the cron entry does not fire. Linux cron keeps no memory of a missed run: if the instance came up at 9:05, the 9:00 job does not run, and nothing reports the miss. That is a Reliability failure, and it is the decisive one here, because it is the exact mechanism behind the Saturday that the client discovered on Monday. It comes paired with an Operational Excellence failure: someone has to patch the host, own a script nobody reviews, and the audit trail of who fired what is whatever the script happens to log. There is a Security failure hanging off the same host whenever the credential arrives as a long-lived access key instead of an instance profile, which is how I have most often found these instances configured, and I count it as a qualified risk rather than a defect of the design, because nothing forces the credential to arrive that way.

The obvious mitigation trades one problem for another. Putting the instance in an Auto Scaling group with capacity 1 fixes availability and introduces duplicate firing during a replacement, when two instances can briefly carry the same crontab. Duplicate firing is the complaint that opened this work.

Cost and sustainability come after, and they are consequence rather than cause. The instance stays up 168 hours a week to run a command that takes less than a second, once a week. The utilization rate is in the order of 0.0001 percent, which is a Cost Optimization failure in the form of a resource billed by time for work measured in milliseconds, and a Sustainability failure in the form of allocated hardware sitting almost entirely idle. Neither of those is the reason to reject the design. They are what the design costs after it already failed on reliability.

An instance starts to earn its place when campaign preparation needs state, a heavy library, or access to something that exists only inside the VPC. That was not the case here. The message body was text.

A fanout subscription on the existing topic

Fanout, in SNS, is the topology of one topic with multiple subscribers receiving the same message in parallel: SQS queues, Lambda functions, HTTP and HTTPS endpoints, email, SMS, mobile push. It is a correct and useful pattern, and it is exactly what this client already ran in production. That is what makes the proposal hard to dismiss on sight: the vocabulary is right.

Fanout is a distribution topology, not a time trigger. An SNS topic is reactive by definition: it distributes what was published to it, at the moment it was published. No topic attribute, subscription attribute or delivery policy introduces time. The question of who publishes on Saturday remains unanswered, so sending stays manual and the original problem does not move. That is an architecture failure in its most direct form, a component chosen that does not address the requirement, and in the vocabulary of the pillars it lands on Operational Excellence, because the manual step stays on the critical path.

One pillar, tied to one mechanism, is the whole analysis here. Reaching for more would be padding.

A Step Functions state machine on a schedule

Before any pillar discussion, the premise has to come apart: AWS Step Functions has no scheduler of its own. The official documentation on starting a state machine execution lists the ways it can happen, and none of them is a scheduler internal to the service: the Step Functions console, a StartExecution API call, Amazon EventBridge reacting to an event, Amazon EventBridge Scheduler for a time-based start, a nested execution from a Task state, and Amazon API Gateway. The service does have wait states, which delay an execution that already started, and that is not scheduling. Step Functions even publishes a page of its own on using EventBridge Scheduler to start a state machine execution at a given time. A pillar does not apply to a capability that does not exist, so this is a factual error, not a design trade-off.

Once the premise is fixed, the design behind the proposal becomes assessable: EventBridge or EventBridge Scheduler triggering a state machine that publishes to SNS. There, pillars apply. A single-step state machine adds a workflow definition, an execution role, an execution history and per-state-transition billing to the path, to do what EventBridge already does by publishing straight to the topic. That is a Cost Optimization failure, since it inserts a component billed per state transition where the native integration bills nothing for the hop, and an Operational Excellence failure, since it adds one more versioned artifact, one more role and one more console to operate without adding a single business step.

The threshold where that changes is worth naming. Step Functions pays for itself when there is real orchestration: branching per audience segment, waiting on human approval, per-step error handling, parallel calls, retry policies that differ by step. Publishing one message is not orchestration.

An EventBridge rule publishing straight to the topic

EventBridge accepts an SNS topic as the target of a rule and performs the publish itself, authorized by the topic resource policy. There is no intermediate function, no code, and no runtime to version. It wins on Operational Excellence, because schedule and authorization are declared in CloudFormation, versioned in Git, and the change history becomes the repository history, with native metrics in the AWS/Events namespace instead of whatever a script chose to log. It wins on Cost Optimization, because no compute resource participates in the scheduling at all, which leaves SNS delivery as the dominant cost. And it wins on Reliability against the other three designs, because there is no host on the path, with a retry policy and a dead letter queue configurable on the target itself.

It is not perfect, and that belongs in the post as much as the win does.

Delivery is at-least-once. For a marketing campaign that means a risk of duplicate email, which is the original complaint. The design lowers the probability without eliminating it.
The cron expression of an AWS::Events::Rule is interpreted in UTC only, with no time zone property. That is where the wrong-hour mistakes come from, and it is the reason a second template exists further down.
The payload arrives in a standard event format. An input transformer fixes that, and the objection is legitimate enough to deserve the mention.
There is no segmentation, no per-recipient content template and no delivery report. Those limits are the subject of the trade-offs section.

Choosing the smallest operational surface

I chose the scheduled rule on the criterion of smallest operational surface: it is the only one of the four designs where nothing new has to be maintained after the deploy. No operating system, no runtime dependency, no build artifact. What remains to operate is a set of alarms and an empty dead letter queue.

I delivered two templates. The first uses AWS::Events::Rule, which is the classic answer and the one the team recognized. The second uses AWS::Scheduler::Schedule, and it is the one I would leave running, for the reason developed further down. The two stacks are alternatives, not complements, and they deploy under different stack names.

The publish path, from schedule to subscriber

EventBridge evaluates the schedule expression and, at the scheduled time, calls sns:Publish on the campaign topic. The topic distributes to the confirmed subscriptions, each protocol at its own pace. If the target invocation fails permanently, the message goes to an SQS dead letter queue with attributes identifying the rule, the target and the error. Four CloudWatch alarms in the rule template watch the places where the flow can break quietly: invocation failure on the EventBridge side, a denied write to the dead letter queue, dead letter queue depth, and delivery failure on the SNS side. The Scheduler template has five, because its trigger side reports the target error and the dropped invocation as separate metrics.

flowchart LR
  RULE[EventBridge scheduled rule] -->|sns:Publish| TOPIC[SNS topic encrypted with KMS]
  TOPIC --> MAIL[Email subscriptions]
  TOPIC --> SMS[SMS subscriptions]
  TOPIC --> PUSH[Mobile push subscriptions]
  TOPIC --> QUEUE[SQS queue for the internal consumer]
  RULE -->|retries exhausted| DLQ[(SQS dead letter queue)]
  RULE -.->|FailedInvocations| CW[CloudWatch alarms]
  RULE -.->|InvocationsFailedToBeSentToDlq| CW
  DLQ -.->|ApproximateNumberOfMessagesVisible| CW
  TOPIC -.->|NumberOfNotificationsFailed| CW

EventBridge cron expressions run in UTC

This detail cost a campaign at an earlier client, so it comes before the template.

The cron expression of an AWS::Events::Rule has six fields and is always evaluated in UTC. There is no time zone property on the resource. The AWS documentation makes it explicit by annotating every example hour with UTC+0.

Field	Values	Wildcards accepted
Minutes	0-59	`,` `-` `*` `/`
Hours	0-23	`,` `-` `*` `/`
Day-of-month	1-31	`,` `-` `*` `?` `/` `L` `W`
Month	1-12 or JAN-DEC	`,` `-` `*` `/`
Day-of-week	1-7 or SUN-SAT	`,` `-` `*` `?` `L` `#`
Year	1970-2199	`,` `-` `*` `/`

The sixth field, year, is not optional. Anyone pasting a five-field crontab gets a validation error.

The second trap sits in the day fields. The documentation does not allow day-of-month and day-of-week in the same expression: if one of them carries a value or *, the other needs ?. That is why the campaign expression is:

cron(0 12 ? * SAT *)

and not cron(0 12 * * SAT *), which is rejected for using * in day-of-month alongside SAT in day-of-week. The ? means any, and it marks the field as not participating in the decision.

On the hour itself: 12:00 UTC corresponds to 09:00 in America/Sao_Paulo, which sits at UTC-3. For this origin zone the risk is not that the offset moves. It is that the conversion exists at all and lives in the operator's head, so the number in the template no longer says what the business asked for, and reading it back correctly depends on someone remembering the offset. Add the subscribers in other countries, where the offset does shift twice a year, and the rule keeps firing at the same absolute instant while the local hour moves under it. Both failures come from the same place: the expression carries no time zone, so the time zone has to be carried by a person.

The CloudFormation template for the EventBridge rule

Full file in iac/eventbridge-sns-notifications.yaml. What follows are the parts that need justification.

Parameter validation that fails the deploy instead of the campaign

Everything that changes between accounts is a parameter, with validation. Three of them carry most of the information. The topic ARN decides whether the stack owns the topic at all and gets a section of its own below. The other two are the key and the schedule expression:

  ExistingKmsKeyArn:
    Type: String
    Default: ''
    AllowedPattern: '^$|^arn:aws[a-zA-Z-]*:kms:[a-z0-9-]+:[0-9]{12}:key/[0-9a-zA-Z-]+$'
    ConstraintDescription: >-
      Leave empty or provide the ARN of a KMS key, in the form
      arn:aws:kms:REGION:ACCOUNT:key/KEY_ID. An alias ARN is rejected on
      purpose: alias/aws/sns has a key policy that cannot be edited, so the
      grant to events.amazonaws.com has nowhere to go.
    Description: >-
      ARN of an existing customer managed KMS key for the topic this stack
      creates. Leave empty and the stack creates its own key. Ignored when
      ExistingTopicArn is set, because that topic keeps whatever key it already
      has. The key you name here has to grant kms:GenerateDataKey* and
      kms:Decrypt to events.amazonaws.com in its own key policy: this stack
      writes a key policy only for the key it creates. Do not point this at
      alias/aws/sns either, because the key policy of an AWS managed key cannot
      be edited, so the grant has nowhere to go and the rule publish fails with
      KMSAccessDeniedException.

The AllowedPattern on that parameter is the part doing real work. The warning in the Description is readable, and a warning in a description stops nobody: pass an alias ARN and the stack reaches CREATE_COMPLETE, then the campaign fails on the first Saturday. The pattern accepts an empty string or a key ARN and rejects anything with alias/ in the resource part, which turns the warning into a parameter validation failure at deploy time. The same parameter in the Scheduler template carries the pattern for a second reason: there the value also lands in the Resource of an IAM statement, and a malformed ARN there fails the role creation with MalformedPolicyDocument, an error that names neither the parameter nor the template line that caused it.

AlarmTopicArn gets the same treatment, accepting an empty string or an SNS topic ARN. That one is a partial guard and worth saying so: the ARN of the campaign topic is itself a valid SNS topic ARN, so the pattern catches a typo and does not catch the mistake that matters, which is pointing the alarm action at the campaign topic.

  ScheduleExpression:
    Type: String
    Default: cron(0 12 ? * SAT *)
    MinLength: 1
    Description: >-
      Schedule expression of the rule. AWS::Events::Rule always evaluates cron
      in UTC, with no time zone option. cron(0 12 ? * SAT *) is Saturday 12:00
      UTC, which is 09:00 in America/Sao_Paulo. The ? in the day-of-month field
      is required whenever the day-of-week field carries a value.

Whether the stack owns the campaign topic

The requirement was that the existing topic keeps its subscriptions. The first version of this template declared an AWS::SNS::Topic and took the name as a parameter, which looks like it honors that requirement and does the opposite, for one reason: sns:CreateTopic is idempotent. The API reference states it plainly, that a requester who already owns a topic with the given name gets that topic's ARN back without a new topic being created.

Follow what CloudFormation does with that. It calls CreateTopic with the name from the parameter. The production topic already exists under that name, so the call succeeds and returns its ARN, and the stack reaches CREATE_COMPLETE holding the production topic as the physical resource of a logical resource it believes it created. Then: the topic policy written by the stack replaces the policy of that topic, so every publisher that depended on the previous one loses access; DisplayName and KmsMasterKeyId overwrite what was there, which changes the sender name on email and the prefix on SMS for the current subscribers; and delete-stack deletes the topic along with the confirmed subscriptions. Nothing in that sequence produces an error. A loud CREATE_FAILED would have been the safe outcome, and it is not what happens.

So ownership is a parameter, not an accident:

  ExistingTopicArn:
    Type: String
    Default: ''
    AllowedPattern: '^$|^arn:aws[a-zA-Z-]*:sns:[a-z0-9-]+:[0-9]{12}:[a-zA-Z0-9_-]{1,256}$'
    ConstraintDescription: >-
      Leave empty or provide the ARN of an SNS topic, in the form
      arn:aws:sns:REGION:ACCOUNT:TOPIC_NAME. The pattern accepts a standard
      topic only. The ARN of a FIFO topic ends in .fifo and is rejected on
      purpose, because a FIFO topic is limited to 100 subscriptions and cannot
      carry this campaign.
    Description: >-
      ARN of the campaign topic that already carries the production
      subscriptions. Leave empty and the stack creates its own topic, which is
      the self-contained lab. With a value, the stack creates no topic and does
      not delete one on teardown, and TopicName, TopicDisplayName and
      ExistingKmsKeyArn are ignored. TestSubscriptionEmail still applies and
      creates a real subscription on the topic you name, so leave it empty here
      unless you want that address on the live campaign. Two prerequisites on
      the topic you name: this stack replaces its access policy, so any
      publisher that depended on the previous policy loses access, and if the
      topic is encrypted its key
      policy already has to grant kms:GenerateDataKey* and kms:Decrypt to
      events.amazonaws.com, because this stack does not write to a key it did
      not create.

  CreateTopic: !Equals [!Ref ExistingTopicArn, '']
  # The stack only writes a key policy for a key it creates, and it only has a
  # topic to attach a key to when it creates the topic. Pointing at an existing
  # topic therefore creates no key, instead of leaving an orphan key behind.
  CreateKmsKey: !And
    - !Condition CreateTopic
    - !Equals [!Ref ExistingKmsKeyArn, '']

Left empty, the stack is a self-contained lab: it stands up a topic equivalent to the production one, wires the schedule to it, and the teardown takes it back out. That is the mode to use while you are still shaping the stack.

With an ARN in it, the topic is outside the stack, which is the point. There is no AWS::SNS::Topic to adopt and no physical resource for delete-stack to remove, so the destructive path closes. The ARN then flows to the Topics and Resource of the topic policy, the rule target, the optional test subscription, the TopicName dimension of the SNS delivery alarm, and the TopicArn output. The alarm dimension is the one that needed thought: it takes a name, not an ARN, and !GetAtt CampaignTopic.TopicName does not exist on this branch, so it comes out of the ARN:

      Dimensions:
        # The AWS/SNS dimension is the topic name, which has to come out of the
        # ARN when the topic was not created here.
        - Name: TopicName
          Value: !If
            - CreateTopic
            - !GetAtt CampaignTopic.TopicName
            - !Select [5, !Split [':', !Ref ExistingTopicArn]]

One thing the parameter does not make safe, and it has to be said rather than buried in a description: pointing at a production topic still replaces that topic's access policy, because AWS::SNS::TopicPolicy replaces instead of adding. Read the existing policy first, with aws sns get-topic-attributes, and carry anything you still need into this template. The parameter removes the deletion path, not the authorization path.

The test subscription deserves the same warning. It is not ignored in this mode, so an address left in TestSubscriptionEmail becomes a real subscription on the production topic, gets a confirmation mail, and from then on receives the actual campaign alongside everyone else. Leave the parameter empty once the topic is the live one.

Encryption at rest, and why the AWS managed key does not work here

My baseline for an SNS topic is encryption at rest with the AWS managed key, alias/aws/sns, with a parameter left open for a customer managed key when compliance asks for one. In this design that baseline does not work, and I found out on the first deploy attempt.

The SNS documentation is direct about it: for an AWS service to publish to an encrypted topic, you need a customer managed key, and you have to grant kms:GenerateDataKey* and kms:Decrypt to the service principal in the key policy. The table of principals lists events.amazonaws.com. The key policy of an AWS managed key cannot be edited, so there is nowhere to put that grant. With alias/aws/sns on the topic, the rule fires, the publish fails, and the error is KMSAccessDeniedException.

There is a second part to it, and it runs against habit: the same page records that aws:SourceAccount, aws:SourceArn and aws:SourceOrgID are not supported in a key policy on the EventBridge to encrypted topic path. The KMS statement stays without an origin condition. That does not open the topic, because who may publish is still controlled by the topic policy, but it is an exception to the reflex of conditioning every service grant.

  CampaignKey:
    Type: AWS::KMS::Key
    Condition: CreateKmsKey
    DeletionPolicy: Retain
    UpdateReplacePolicy: Retain
    Properties:
      Description: !Sub 'Encryption at rest for the campaign topic of stack ${AWS::StackName}'
      EnableKeyRotation: true
      KeyPolicy:
        Version: '2012-10-17'
        Statement:
          - Sid: AllowAccountIamPolicies
            Effect: Allow
            Principal:
              AWS: !Sub 'arn:${AWS::Partition}:iam::${AWS::AccountId}:root'
            Action: 'kms:*'
            Resource: '*'
          # No Condition here on purpose: the SNS documentation records that
          # aws:SourceArn, aws:SourceAccount and aws:SourceOrgID are not
          # supported in a key policy on the EventBridge to encrypted topic
          # path. Who may publish stays controlled by the topic policy below.
          - Sid: AllowEventBridgeToEncryptMessages
            Effect: Allow
            Principal:
              Service: events.amazonaws.com
            Action:
              - 'kms:GenerateDataKey*'
              - 'kms:Decrypt'
            Resource: '*'

That grant is in the key policy of the key this stack creates, which means it covers one of the ways the parameter can be used and not the other. Pass ExistingKmsKeyArn and no key policy is written by the stack at all: the topic uses a key whose policy the template does not control and cannot read. The AllowedPattern rejects an alias ARN and says nothing about the contents of a key policy, so a customer managed key without the grant to events.amazonaws.com passes validation, the stack reaches CREATE_COMPLETE, and the first Saturday reproduces the same KMSAccessDeniedException this section is about. The diagnosis is in the dead letter queue, under ERROR_CODE, as ACTION_DECRYPTION_FAILURE. If you supply a key, read its policy before you supply it.

The key carries DeletionPolicy: Retain on purpose, and the teardown section says what that means in practice.

The topic policy, and what the origin conditions prevent

Authorization for EventBridge to publish is a resource policy on the topic, with principal events.amazonaws.com and action sns:Publish. The documentation example for this case comes with no condition at all, and a template that copies that example literally grants publish rights to any EventBridge rule in any AWS account. That is the confused deputy problem in its most direct form: the service holds the permission, and your resource does not check on whose behalf the service is acting.

The EventBridge documentation on the subject says which value to use: for a target of a rule on an event bus, aws:SourceArn has to be the ARN of the rule. I add aws:SourceAccount to tie it to this account.

The policy has one statement, and the decision to keep it at one is the subject of the paragraphs after the block.

  CampaignTopicPolicy:
    Type: AWS::SNS::TopicPolicy
    Properties:
      Topics:
        - !If [CreateTopic, !Ref CampaignTopic, !Ref ExistingTopicArn]
      PolicyDocument:
        Version: '2012-10-17'
        Statement:
          # Without the two conditions below, any EventBridge rule in any
          # account could publish here. aws:SourceArn ties the permission to
          # this rule; aws:SourceAccount ties it to this account. The SNS
          # access control documentation lists Amazon EventBridge among the
          # services that set SourceAccount when publishing to a topic, so both
          # keys are in the request context on this path.
          - Sid: PublishOnlyFromTheScheduledRule
            Effect: Allow
            Principal:
              Service: events.amazonaws.com
            Action: 'sns:Publish'
            Resource: !If [CreateTopic, !Ref CampaignTopic, !Ref ExistingTopicArn]
            Condition:
              ArnEquals:
                'aws:SourceArn': !GetAtt CampaignRule.Arn
              StringEquals:
                'aws:SourceAccount': !Ref AWS::AccountId

Both condition keys have to be in the request context for that statement to grant anything, and that is worth checking rather than assuming. The SNS access control documentation lists the services that set SourceAccount when they publish to a topic, and Amazon EventBridge is on the list. The same page records that aws:SourceOwner, which appears in older examples for the same purpose, is deprecated, and that new services integrate through aws:SourceArn and aws:SourceAccount.

AWS::SNS::TopicPolicy replaces the topic policy rather than adding to it, which is the fact that makes the single statement a decision instead of an omission. The resulting document is exactly what is written above. There is no administrative statement, no statement for the operations team, and no statement for the account.

I wrote one in the first version of this template, and then took it out. The reasoning behind writing it goes like this: the policy replaces the default, the default grants the account owner the administrative actions, so the replacement should restore them. Two things are wrong with that chain. The fact in the middle goes first: the SNS documentation describes a new topic as created with no permissions, reachable by the account that created it, which is not a default that hands administration to the account through the topic policy, so there is nothing of that kind to restore. The decisive error is the assumption the conclusion rests on and never states. A resource policy grants access on top of whatever the caller identity policy already allows, not instead of it. An administrative statement with principal * and a condition tying it to this account grants those actions to every identity in the account, including a role whose identity policy carries no SNS permission at all. With sns:DeleteTopic, sns:SetTopicAttributes and sns:Subscribe in the list, any internal principal could drop the campaign topic, rewrite this policy or attach an endpoint of its choosing to a topic with millions of subscriptions. sns:AddPermission is the worst of them, because it is the action that adds a cross-account Allow for sns:Publish, which is the exact requirement the design was supposed to hold. No principal outside the account can publish today, and that statement let any principal inside the account change it.

Administration of a topic in the same account needs no statement in the resource policy. The identity policy of the caller authorizes it alone. Manual publishing works the same way, and the validation section comes back to that, because it changes how a failed test publish gets diagnosed.

There is a second trap in the vicinity, and it is about which condition key belongs on which statement. The IAM global condition key reference describes aws:SourceAccount as entering the request context when the call to the resource is made by an AWS service principal, carrying the account the service acts on behalf of. It describes aws:PrincipalAccount as present in every request, anonymous ones included, comparing the account the calling principal belongs to. Two keys with similar names, two different request contexts. The publish statement above is called by a service principal, which is where aws:SourceAccount applies. A statement written for a call made directly by an IAM principal is a different context and reaches for aws:PrincipalAccount instead.

What makes that easy to get backwards is the shape of the policy people copy from. The EventBridge documentation on resource-based policies reproduces a default SNS topic policy whose single statement has Principal as * and conditions the administrative actions on AWS:SourceAccount. Lifting that shape into a hand-written policy is how a statement ends up resting on a key nobody verified for that path. For the case here the SNS access control documentation closes the question from the other side: a new topic is created private, with no permissions, reachable by the account that created it. Administration inside the same account needs no statement on the topic. It needs an identity policy.

The input transformer and the execution identifier

The default scheduled event reaches the subscriber as the metadata envelope every EventBridge event carries, with the fields version, id, detail-type, source, account, time, region, resources and detail. For a scheduled rule the service fills it the way the events reference documents, with the ARN of the rule in resources and detail empty, because a firing has nothing to describe:

{
  "version": "0",
  "id": "7c4b21f0-9e3a-4b55-8f21-2c6d9a4f1b8e",
  "detail-type": "Scheduled Event",
  "source": "aws.events",
  "account": "123456789012",
  "time": "2026-10-10T12:00:00Z",
  "region": "us-east-1",
  "resources": ["arn:aws:events:us-east-1:123456789012:rule/weekly-campaigns-rule-saturday-trigger"],
  "detail": {}
}

The time is the firing this rule produces: Saturday, October 10, 2026, 12:00 UTC, 09:00 in America/Sao_Paulo. The id is a UUID that EventBridge generates per event. Useful for debugging, useless as a campaign, since what the subscriber would read is the envelope rather than a message. The input transformer replaces the payload.

          InputTransformer:
            InputPathsMap:
              eventId: '$.id'
              scheduledTime: '$.time'
            InputTemplate: |
              {
                "campaign": "weekly-newsletter",
                "executionId": "<eventId>",
                "scheduledTime": "<scheduledTime>",
                "ingestionTime": "<aws.events.event.ingestion-time>",
                "rule": "<aws.events.rule-name>"
              }

InputPathsMap declares variables from JSONPath over the original event. aws.events.event.ingestion-time and aws.events.rule-name are predefined variables: they do not need declaring, and the first one is generated by EventBridge as an ISO 8601 timestamp and cannot be overwritten. A variable pointing at a scalar value goes inside quotes in the template; a variable pointing at an object or array does not.

Against the event above, the template renders the body the topic publishes:

{
  "campaign": "weekly-newsletter",
  "executionId": "7c4b21f0-9e3a-4b55-8f21-2c6d9a4f1b8e",
  "scheduledTime": "2026-10-10T12:00:00Z",
  "ingestionTime": "2026-10-10T12:00:03Z",
  "rule": "weekly-campaigns-rule-saturday-trigger"
}

The two timestamps are not the same instant, and the gap is the design rather than a defect: scheduledTime is the minute the schedule fired, ingestionTime is when EventBridge received the event, and the documentation is explicit that several seconds can pass between the scheduled time and the target running. A consumer that treats them as interchangeable will eventually file a firing under the wrong minute.

The executionId field carries the event id, and it exists because of the idempotency discussion further down. Retries of the same event keep the same id, while each weekly firing produces a new one. That is the key a consumer uses to recognize an occurrence it already processed.

One consequence has to be clear: plain email subscribers receive this JSON as the message body. For a campaign with formatted content, either the InputTemplate emits a string instead of JSON, or the topic feeds a consumer that renders the content. The JSON envelope serves the design where there is a consumer on the other side.

Explicit retry policy and dead letter queue

EventBridge defaults to retrying for 24 hours and up to 185 attempts, with exponential backoff and jitter. For a campaign with a fixed hour, that default has an unwanted effect: a long outage results in delivery the next day, which for marketing is worse than no delivery. I cut the window to one hour and send the rest to the dead letter queue.

          RetryPolicy:
            MaximumEventAgeInSeconds: 3600
            MaximumRetryAttempts: 10
          DeadLetterConfig:
            Arn: !GetAtt CampaignDlq.Arn

The dead letter queue is not decoration. Some errors never go through a retry at all: missing permission on the target, a target that no longer exists, an address resolution failure. The documentation says those go straight to the dead letter queue when one is configured. Without it, the only clue would be the FailedInvocations metric, which tells you a failure happened and carries neither the payload nor the reason. Each message in the queue arrives with the attributes RULE_ARN, TARGET_ARN, ERROR_CODE, ERROR_MESSAGE, EXHAUSTED_RETRY_CONDITION and RETRY_ATTEMPTS.

Two documented constraints on an EventBridge dead letter queue: standard queues only, FIFO is not accepted, and the queue has to be in the same region as the rule.

Queue retention is the parameter I nearly left at the default. MessageRetentionPeriod accepts from 60 seconds to 1,209,600 seconds, which is 14 days, and the default is 345,600 seconds, 4 days. On a weekly campaign, 4 days is short: a Saturday failure expires on Wednesday, before anyone has a reason to look. The template uses the maximum.

The permission to write to the queue deserves a warning. When you configure a dead letter queue through the console, the console attaches the resource policy to the queue for you. Through the API or CloudFormation, it does not. It is a common and quiet deploy error: the stack comes up, the rule fires, the invocation fails, and the attempt to write to the dead letter queue fails too. The symptom shows up in InvocationsFailedToBeSentToDlq, and the queue depth alarm says nothing, because in that failure mode nothing ever reaches the queue.

  CampaignDlqPolicy:
    Type: AWS::SQS::QueuePolicy
    Properties:
      Queues:
        - !Ref CampaignDlq
      PolicyDocument:
        Version: '2012-10-17'
        Statement:
          # Only aws:SourceArn here, matching the target dead letter queue
          # example in the EventBridge documentation. The rule ARN already
          # carries the account ID, so the account pin is not lost, and this
          # path has no published confirmation that aws:SourceAccount reaches
          # the request context. An Allow conditioned on a key that is absent
          # grants nothing, and since AWS::SQS::QueuePolicy replaces the queue
          # policy, that would silently take the dead letter path offline.
          - Sid: AllowEventBridgeToWriteToTheDlq
            Effect: Allow
            Principal:
              Service: events.amazonaws.com
            Action: 'sqs:SendMessage'
            Resource: !GetAtt CampaignDlq.Arn
            Condition:
              ArnEquals:
                'aws:SourceArn': !GetAtt CampaignRule.Arn

That statement carries one condition key, and the topic policy a few paragraphs up carries two. The asymmetry is deliberate. For the publish path the SNS documentation names EventBridge as a service that sets SourceAccount, so both keys are confirmed present. For a write to a target dead letter queue the official example in the EventBridge dead letter queue documentation uses only aws:SourceArn, and I did not find published confirmation that aws:SourceAccount reaches the request context on that path. Adding a key on a guess has an asymmetric cost here. An Allow whose condition cannot match grants nothing, AWS::SQS::QueuePolicy replaces the queue policy rather than adding to it, and the result would be the exact quiet failure described above, with the whole evidence trail gone. The rule ARN already contains the account ID, so pinning the account is not something the shorter statement gives up.

Four alarms and the TreatMissingData detail

  RuleFailedInvocationsAlarm:
    Type: AWS::CloudWatch::Alarm
    Properties:
      AlarmName: !Sub '${AWS::StackName}-failed-invocations'
      AlarmDescription: The scheduled rule could not publish to the SNS topic
      Namespace: AWS/Events
      MetricName: FailedInvocations
      Dimensions:
        - Name: RuleName
          Value: !Ref CampaignRule
      Statistic: Sum
      Period: 300
      EvaluationPeriods: 1
      Threshold: 0
      ComparisonOperator: GreaterThanThreshold
      TreatMissingData: notBreaching

FailedInvocations is sent to CloudWatch only when it is non-zero. On a healthy weekly rule, the metric spends the whole week without a single data point. With the default TreatMissingData, the alarm lives in INSUFFICIENT_DATA and nobody trusts it. notBreaching reads the absence of data as success, which is the correct semantics for this metric. Every alarm in both templates sets it explicitly for the same reason.

The second alarm watches ApproximateNumberOfMessagesVisible on the dead letter queue, because a message sitting there is a firing that was lost with evidence attached.

The third one is there because the first two give the same answer to two different questions. FailedInvocations fires whether the evidence was preserved or discarded, and the depth alarm stays at zero in the case where the write to the queue was denied, since nothing arrived. The operator ends up with one signal covering both "it failed, go read the queue" and "it failed, there is nothing to read", and that difference decides whether the campaign can be reprocessed. InvocationsFailedToBeSentToDlq, in AWS/Events with the RuleName dimension, separates them:

  RuleDlqWriteFailureAlarm:
    Type: AWS::CloudWatch::Alarm
    Properties:
      AlarmName: !Sub '${AWS::StackName}-dlq-write-failed'
      AlarmDescription: EventBridge could not write a failed invocation to the dead letter queue
      Namespace: AWS/Events
      MetricName: InvocationsFailedToBeSentToDlq
      Dimensions:
        - Name: RuleName
          Value: !Ref CampaignRule
      Statistic: Sum
      Period: 300
      EvaluationPeriods: 1
      Threshold: 0
      ComparisonOperator: GreaterThanThreshold
      TreatMissingData: notBreaching

Watch the capitalization. A metric name is case sensitive, the spelling in the metric table of the EventBridge monitoring page is InvocationsFailedToBeSentToDlq, and prose elsewhere tends to write DLQ in capitals. An alarm built on the wrong spelling never leaves INSUFFICIENT_DATA and nothing tells you why.

The fourth one exists because the first three only see the trigger. A successful publish with delivery failing at scale lights up none of them, and that was the blind spot I wanted closed. NumberOfNotificationsFailed in the AWS/SNS namespace, with dimension TopicName, covers the delivery side. Its threshold is a parameter, DeliveryFailureThreshold, rather than a hardcoded zero, because the metric counts differently per protocol: it increments by 1 per message when SNS gives up on SQS, email, SMS and push, but counts every failed attempt on HTTP and HTTPS, retries included. On a topic with HTTP or HTTPS subscriptions, a transient endpoint problem produces data points with no definitive loss, and a threshold of zero would page someone for nothing. The ceiling for a single undeliverable message is the numRetries of the delivery policy on that subscription, which defaults to 3 and accepts up to 100, so the threshold is a value you derive from the retry counts on the topic rather than pick by feel. The trade-offs section works through those policies per endpoint type.

One more parameter with a reason behind it: AlarmTopicArn. Pointing the alarm action at the campaign topic would turn every alarm into a notification to millions of subscribers. The alarm topic is an operations topic and has to be a different one. Left empty, the alarms are created with no action, which is what you want while the stack is still being shaped.

Deploying the stack

Example values used from here on: <STACK_NAME> is weekly-campaigns-rule, <SCHEDULER_STACK_NAME> is weekly-campaigns-scheduler, <REGION> is sa-east-1, <ACCOUNT_ID> is 123456789012, the account ID the AWS documentation uses in its own examples and the one every ARN below carries in place of a real account. Two more placeholders appear in the metric read later on, <START_TIME> and <END_TIME>, both in the format 2026-10-08T00:00:00Z, and they are not fixed values: they bound the window of the test you just ran, not the date of this post. The two stack names are deliberate: the templates are alternatives, and they compose physical names for the queue, the key alias and the alarms from the stack name, so they are meant to live under distinct names rather than replace each other.

Those two stack names resolve to the following physical names and ARNs. Having the table in front of you beats resolving a !Sub in your head while reading a command:

Resource	Physical name	ARN
Campaign topic, rule stack	`weekly-campaign-subscribers`	`arn:aws:sns:sa-east-1:123456789012:weekly-campaign-subscribers`
Scheduled rule	`weekly-campaigns-rule-saturday-trigger`	`arn:aws:events:sa-east-1:123456789012:rule/weekly-campaigns-rule-saturday-trigger`
Rule dead letter queue	`weekly-campaigns-rule-dlq`	`arn:aws:sqs:sa-east-1:123456789012:weekly-campaigns-rule-dlq`
Rule topic key alias	`alias/weekly-campaigns-rule-topic-key`	`arn:aws:kms:sa-east-1:123456789012:alias/weekly-campaigns-rule-topic-key`
Rule invocation failure alarm	`weekly-campaigns-rule-failed-invocations`	`arn:aws:cloudwatch:sa-east-1:123456789012:alarm:weekly-campaigns-rule-failed-invocations`
Campaign topic, Scheduler stack	`weekly-campaign-subscribers-scheduler`	`arn:aws:sns:sa-east-1:123456789012:weekly-campaign-subscribers-scheduler`
Schedule group	`weekly-campaigns-scheduler-group`	`arn:aws:scheduler:sa-east-1:123456789012:schedule-group/weekly-campaigns-scheduler-group`
Schedule	`weekly-campaigns-scheduler-saturday-trigger`	`arn:aws:scheduler:sa-east-1:123456789012:schedule/weekly-campaigns-scheduler-group/weekly-campaigns-scheduler-saturday-trigger`
Schedule dead letter queue	`weekly-campaigns-scheduler-dlq`	`arn:aws:sqs:sa-east-1:123456789012:weekly-campaigns-scheduler-dlq`

Three of those shapes are where a hand-written ARN goes wrong. A schedule ARN carries the group before the name, schedule/GROUP/NAME, while the group is a resource of its own, schedule-group/GROUP, and pinning the wrong one in the trust policy condition is the mistake the Scheduler section comes back to. An alarm ARN separates the name with a colon, alarm:NAME, not a slash. And a queue has two identifiers rather than one: the ARN above, which is what the policies and the DeadLetterConfig take, and the URL https://sqs.sa-east-1.amazonaws.com/123456789012/weekly-campaigns-rule-dlq, which is what receive-message takes and what the DlqUrl output returns.

The Scheduler execution role is the one name absent from that table, and that is deliberate in the template: it declares no RoleName, so CloudFormation composes the physical name from the stack name, the logical ID and a random suffix, giving something shaped like weekly-campaigns-scheduler-CampaignSchedulerRole-<SUFFIX>. Naming it would mean CAPABILITY_NAMED_IAM instead of CAPABILITY_IAM in every deploy below, and a collision the first time the same stack goes to a second region, because IAM is global. Read the ARN off the SchedulerRoleArn output rather than reconstructing it.

TopicName is the one physical name that does not derive from the stack name, and it is a parameter with a fixed default on purpose, because the topic ARN is what the existing subscriptions and the other stacks depend on. That makes it the parameter to check before a deploy, for the reason covered in the section on topic ownership: sns:CreateTopic is idempotent, so CloudFormation gives you no protection against reusing a name. Deploy the same template twice in the same region under two stack names and the second stack does not fail. It succeeds, adopts the same physical topic, replaces the topic policy the first stack wrote, overwrites the display name and the key, and then deletes that topic on whichever delete-stack runs first, while the other stack goes on reporting CREATE_COMPLETE for a resource that is gone.

The check is a command, not a hope for CREATE_FAILED. Call aws sns get-topic-attributes on the ARN the parameter would produce, and let a NotFound be the green light:

aws sns get-topic-attributes \
  --topic-arn arn:aws:sns:<REGION>:<ACCOUNT_ID>:weekly-campaign-subscribers \
  --region <REGION> \
  --profile <PROFILE>

Expected result on a free name: NotFound. The code and the exception class differ, which matters if you script this check: the CLI prints the response code, NotFound, while the SDKs expose the same error as a class named NotFoundException. Match on the response code. Anything else means the name is taken, and the choice is then to pick another name, deploy to another region, or pass the ARN in ExistingTopicArn on purpose. The two templates in this post do not share a topic, and the reason is that their defaults differ, not that CloudFormation would have stopped them.

Lint first. If it is not installed:

pip install cfn-lint

cfn-lint iac/eventbridge-sns-notifications.yaml
cfn-lint iac/eventbridge-scheduler-sns.yaml

Expected result: empty output and return code 0 on both files, meaning no findings. Then the service-side check, which requires credentials:

aws cloudformation validate-template \
  --template-body file://iac/eventbridge-sns-notifications.yaml \
  --region <REGION> \
  --profile <PROFILE>

Expected result: the service returns the parameter list, the template description and the capabilities it requires. Without credentials configured, this command fails on credentials rather than on the template, which is a different outcome from a template problem and should not be read as one.

Deploy. --capabilities CAPABILITY_IAM is required by the Scheduler template, which creates a role, and harmless on this one:

aws cloudformation deploy \
  --template-file iac/eventbridge-sns-notifications.yaml \
  --stack-name <STACK_NAME> \
  --parameter-overrides \
      TopicName=weekly-campaign-subscribers \
      ScheduleExpression='cron(0 12 ? * SAT *)' \
      RuleState=ENABLED \
      TestSubscriptionEmail=ops@example.com \
      ProjectTag=weekly-campaigns \
  --capabilities CAPABILITY_IAM \
  --region <REGION> \
  --profile <PROFILE>

Outputs:

aws cloudformation describe-stacks \
  --stack-name <STACK_NAME> \
  --query 'Stacks[0].Outputs' \
  --output table \
  --region <REGION> \
  --profile <PROFILE>

Expected result: TopicArn, RuleName, RuleArn, DlqUrl, DlqArn and KmsKeyArn. The TopicArn has the form arn:aws:sns:sa-east-1:123456789012:weekly-campaign-subscribers and is the <TOPIC_ARN> of the commands below, and DlqUrl is the <DLQ_URL>.

Inspecting what was created:

aws events describe-rule \
  --name weekly-campaigns-rule-saturday-trigger \
  --region <REGION> \
  --profile <PROFILE>

aws events list-targets-by-rule \
  --rule weekly-campaigns-rule-saturday-trigger \
  --region <REGION> \
  --profile <PROFILE>

Expected result: the first confirms ScheduleExpression and State: ENABLED. The second is the one I read with more attention, because it shows the target Arn, the InputTransformer with the rendered template, and the DeadLetterConfig. If DeadLetterConfig is not there, the dead letter queue is not wired, whatever the template says.

Proving the schedule works without waiting for Saturday

Three checks, in increasing order of isolation.

The first separates delivery failure from scheduling failure. Publish to the topic by hand and confirm the message reaches the test endpoint:

aws sns publish \
  --topic-arn <TOPIC_ARN> \
  --message '{"campaign":"weekly-newsletter","executionId":"manual-test-001"}' \
  --subject 'Delivery test' \
  --region <REGION> \
  --profile <PROFILE>

Expected result: a MessageId comes back. Two permissions decide this command, and both matter for diagnosis. The topic policy grants sns:Publish only to the EventBridge service principal, so an operator publishing by hand is authorized by the IAM policy of their own identity, not by the topic policy. On top of that, the topic is encrypted, so the same identity needs kms:GenerateDataKey* and kms:Decrypt on the topic key. A KMS error here points at the key policy or the caller KMS permissions; an SNS authorization error points at the caller IAM policy.

The second exercises the schedule itself by shortening the interval. This changes the stack and has to be reverted:

# WARNING: temporary change. Revert to cron(0 12 ? * SAT *) after the test.
aws cloudformation deploy \
  --template-file iac/eventbridge-sns-notifications.yaml \
  --stack-name <STACK_NAME> \
  --parameter-overrides ScheduleExpression='rate(5 minutes)' \
  --capabilities CAPABILITY_IAM \
  --region <REGION> \
  --profile <PROFILE>

Expected result: within fifteen minutes there are enough data points for the metric read below. Reverting is a deploy with the original expression. Leaving rate(5 minutes) forgotten on a production topic is the kind of accident that turns into a real incident, so the revert is part of the test and not a follow-up.

The third is the metric read. Invocations in the AWS/Events namespace counts the times the target was invoked. Set the window to the minutes you spent on the rate(5 minutes) test, because a window that misses it returns an empty list for the wrong reason, and an empty list is also what the healthy path returns on the failure metric below:

aws cloudwatch get-metric-statistics \
  --namespace AWS/Events \
  --metric-name Invocations \
  --dimensions Name=RuleName,Value=weekly-campaigns-rule-saturday-trigger \
  --start-time <START_TIME> \
  --end-time <END_TIME> \
  --period 300 \
  --statistics Sum \
  --region <REGION> \
  --profile <PROFILE>

Expected result: one data point with Sum: 1 per firing inside the window. Swapping --metric-name for FailedInvocations, the expected result is Datapoints: [], because that metric is published only when it is non-zero. An empty list there is the good outcome.

If something did fail, the dead letter queue holds the detail:

aws sqs receive-message \
  --queue-url <DLQ_URL> \
  --message-attribute-names All \
  --max-number-of-messages 10 \
  --region <REGION> \
  --profile <PROFILE>

Expected result: the ERROR_CODE attribute directs the diagnosis. NO_PERMISSIONS points at the topic policy or the key policy. ACTION_DECRYPTION_FAILURE points at KMS.

Teardown, which is never optional:

aws cloudformation delete-stack --stack-name <STACK_NAME> --region <REGION> --profile <PROFILE>
aws cloudformation wait stack-delete-complete --stack-name <STACK_NAME> --region <REGION> --profile <PROFILE>

That removes the topic only when the stack created it. With ExistingTopicArn set there is no topic in the stack, so the teardown leaves the topic and the subscriptions it already had in place and takes away the rule, the queue, the alarms and the topic policy. One subscription does go, if TestSubscriptionEmail was filled in on that deploy: the stack owns the test subscription even when it does not own the topic, and that is the only subscription of an external topic the teardown touches. The policy is the part to plan for: deleting the stack removes the statement that authorized the rule, and it does not restore whatever policy the topic had before the stack replaced it.

The KMS key carries DeletionPolicy: Retain, so it survives the delete-stack and has to be scheduled for deletion separately. That is deliberate: a key deleted along with the stack makes anything encrypted under it unreadable. The alias does not survive, which is what lets you recreate the stack under the same name without a collision.

The same thing happens on an update, and it surprises people, so it is worth tracing. Supply ExistingKmsKeyArn or ExistingTopicArn to a stack that was created with the defaults and CreateKmsKey turns false, which takes both the key and its alias out of the stack. The key is retained and keeps being billed as a key, with rotation still enabled; the alias has no DeletionPolicy and is deleted. So the key you have to clean up afterwards no longer has a name pointing at it, and aws kms schedule-key-deletion needs the key ID. Read it off the KmsKeyArn output before the update, not after.

ExistingTopicArn takes a second resource out of the stack on that same update, and this one is the intended path rather than an accident. Passing it turns CreateTopic false, and CampaignTopic carries no DeletionPolicy, so the topic the stack had created is deleted, taking its subscriptions with it. That is the transition from the self-contained lab to the production topic, and discarding the lab topic is the cleanup you want at that moment. Clear TestSubscriptionEmail in the same update, though. Left filled, the test subscription is not deleted but recreated against the ARN you just supplied, which is the one way these stacks subscribe an address to the production topic. If you would rather keep the lab topic after the switch, DeletionPolicy: Retain on CampaignTopic is the edit, and then that topic also survives the teardown and joins the KMS key on the list of things to clean up by hand.

The EventBridge Scheduler version, with a native time zone

The EventBridge documentation classifies scheduled rules as a legacy feature and recommends EventBridge Scheduler for invoking targets on a schedule. For this case the difference that matters is one property: ScheduleExpressionTimezone.

Full file in iac/eventbridge-scheduler-sns.yaml.

  CampaignSchedule:
    Type: AWS::Scheduler::Schedule
    Properties:
      Name: !Sub '${AWS::StackName}-saturday-trigger'
      Description: Weekly Saturday campaign, evaluated in the local time zone
      GroupName: !Ref CampaignScheduleGroup
      State: !Ref ScheduleState
      ScheduleExpression: !Ref ScheduleExpression
      ScheduleExpressionTimezone: !Ref ScheduleTimezone
      FlexibleTimeWindow: !If
        - UseFlexibleWindow
        - Mode: FLEXIBLE
          MaximumWindowInMinutes: !Ref FlexibleWindowMinutes
        - Mode: 'OFF'
      Target:
        # Templated Scheduler target for Amazon SNS Publish: the topic ARN is
        # enough, with no service specific parameter block.
        Arn: !If [CreateTopic, !GetAtt CampaignTopic.TopicArn, !Ref ExistingTopicArn]
        RoleArn: !GetAtt CampaignSchedulerRole.Arn
        # Scheduler context attributes. execution-id is assigned per invocation
        # attempt, so it does not work as a deduplication key: whoever
        # deduplicates uses scheduled-time, which is stable across the attempts
        # of one occurrence. An attempt-number above 1 identifies a retry.
        Input: |
          {
            "campaign": "weekly-newsletter",
            "executionId": "<aws.scheduler.execution-id>",
            "attempt": "<aws.scheduler.attempt-number>",
            "scheduledTime": "<aws.scheduler.scheduled-time>",
            "schedule": "<aws.scheduler.schedule-arn>"
          }
        RetryPolicy:
          MaximumEventAgeInSeconds: 3600
          MaximumRetryAttempts: 10
        DeadLetterConfig:
          Arn: !GetAtt CampaignDlq.Arn

The expression becomes cron(0 9 ? * SAT *) with ScheduleExpressionTimezone: America/Sao_Paulo, and the 9 means nine in the morning local time. The conversion leaves the operator's head and becomes the service's responsibility. For a client firing campaigns in several countries, that means one schedule per time zone instead of a conversion spreadsheet.

The same Saturday firing, with the context attributes resolved by Scheduler instead of by an input transformer, reaches the topic like this:

{
  "campaign": "weekly-newsletter",
  "executionId": "c7a4k91fe2db5a08",
  "attempt": "1",
  "scheduledTime": "2026-10-10T12:00:00Z",
  "schedule": "arn:aws:scheduler:sa-east-1:123456789012:schedule/weekly-campaigns-scheduler-group/weekly-campaigns-scheduler-saturday-trigger"
}

Two details in there are worth more than the shape. The executionId is not a UUID: the documented example of aws.scheduler.execution-id is a short alphanumeric string, assigned per invocation attempt, which is the second reason the idempotency section refuses it as a deduplication key. And scheduledTime is written as the UTC instant, which is the documented example format for the attribute, with the Z suffix, rather than something I read off a capture. The schedule fires at 09:00 local and that is the same moment as 12:00 UTC, so a consumer that formats the field for a human has to apply the time zone back. If your pipeline branches on that value, confirm it on your own schedule before trusting either reading.

Mode: 'OFF' needs the quotes in YAML, otherwise the parser reads OFF as boolean false. MaximumWindowInMinutes accepts 1 to 1440. I left FLEXIBLE as the default with 5 minutes, because spreading the publish inside a window smooths the spike on the consumer side. A campaign with a contractual cut-off time uses OFF.

Three structural differences between the two templates.

Authorization moves. The rule publishes as a service principal, authorized by the topic resource policy. Scheduler assumes a role and publishes as that role. That takes the KMS complication out of the path: the key policy can delegate to IAM in the ordinary way, and the grant of kms:GenerateDataKey* and kms:Decrypt goes in the role policy with the key ARN as the resource. The same role gets sqs:SendMessage on the dead letter queue, which is why that template has no AWS::SQS::QueuePolicy. One permission path instead of two.

That also lifts the constraint that forced a customer managed key in the other template. The SNS documentation puts the role publishing to an encrypted topic under the ordinary case, where the publisher carries kms:GenerateDataKey* and kms:Decrypt in its own identity policy, so alias/aws/sns would work here and my usual baseline would apply. This template creates a key anyway, for two reasons worth naming because they are choices and not requirements: an editable key policy is what makes the delegation below visible as a statement you can read, and keeping the key parameter identical across the two stacks means the teardown and the retained-key caveat read the same way in both. Swap it for the AWS managed key and the only thing you lose is that discussion.

The delegation is doing load-bearing work in that sentence, and it only holds for the key the stack creates. KMS evaluates the key policy and the identity policy together, so an IAM grant on a key whose policy neither delegates to the account root nor names this role authorizes nothing. The key this template creates carries the delegation statement; a key supplied through ExistingKmsKeyArn may not. That is also why supplying an existing topic does not make the key parameter pointless here, unlike in the rule template: with an external encrypted topic, ExistingKmsKeyArn is how the role gets access to the key that topic already uses, and the statement is dropped from the role policy altogether when there is no key to name.

The trust policy gains origin conditions. Scheduler supports aws:SourceArn pointing at the schedule group, and that is why the template creates a dedicated AWS::Scheduler::ScheduleGroup instead of using the default group.

      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service: scheduler.amazonaws.com
            Action: 'sts:AssumeRole'
            # Without these conditions the role can be assumed on behalf of any
            # schedule in the service, including one in another account, which
            # is the confused deputy problem.
            Condition:
              StringEquals:
                'aws:SourceAccount': !Ref AWS::AccountId
              ArnEquals:
                'aws:SourceArn': !Sub 'arn:${AWS::Partition}:scheduler:${AWS::Region}:${AWS::AccountId}:schedule-group/${CampaignScheduleGroup}'

The metrics lose granularity, and that is the price. The AWS/Scheduler namespace has a single dimension, ScheduleGroup. There is no per-schedule dimension. A group with thirty schedules produces alarms that say something in the group failed without saying what. The dedicated group per campaign is what keeps the alarm specific, and it is a modeling cost the rule does not charge, since AWS/Events has a RuleName dimension. This template carries five alarms instead of four, because the trigger side reports TargetErrorCount and InvocationDroppedCount as separate signals: the first is the target returning an error on the API call, the second is Scheduler exhausting the retry policy and dropping the invocation. The dead letter queue pair is the same idea as in the rule template, with the names Scheduler uses: InvocationsFailedToBeSentToDeadLetterCount for the denied write, and queue depth for the write that worked.

Deploy and inspection, under the scheduler stack name:

aws cloudformation deploy \
  --template-file iac/eventbridge-scheduler-sns.yaml \
  --stack-name <SCHEDULER_STACK_NAME> \
  --parameter-overrides \
      ScheduleExpression='cron(0 9 ? * SAT *)' \
      ScheduleTimezone=America/Sao_Paulo \
      ProjectTag=weekly-campaigns \
  --capabilities CAPABILITY_IAM \
  --region <REGION> \
  --profile <PROFILE>

aws scheduler get-schedule \
  --name weekly-campaigns-scheduler-saturday-trigger \
  --group-name weekly-campaigns-scheduler-group \
  --region <REGION> \
  --profile <PROFILE>

Expected result: the schedule comes back with ScheduleExpressionTimezone set, the FlexibleTimeWindow block, and the target carrying RoleArn, Input, RetryPolicy and DeadLetterConfig.

This stack gets torn down the same way, under its own name:

aws cloudformation delete-stack --stack-name <SCHEDULER_STACK_NAME> --region <REGION> --profile <PROFILE>
aws cloudformation wait stack-delete-complete --stack-name <SCHEDULER_STACK_NAME> --region <REGION> --profile <PROFILE>

That removes the queue, the schedule and the schedule group that contains it, the execution role and the five alarms, plus the topic when the stack created it. One subscription goes with them, if TestSubscriptionEmail was filled in on that deploy: the stack owns the test subscription even when it does not own the topic, and that is the only subscription of an external topic the teardown touches. The KMS key of this stack carries DeletionPolicy: Retain as well, so it survives here too and needs its own aws kms schedule-key-deletion, by key ID once the alias is gone. Recreating the stack under the same name creates a new key, and the retained ones accumulate per cycle, each one billed as a key.

Trade-offs, quotas and production concerns

Delivering to millions of subscribers

The schedule publishes one message a week. All the scale lives on the SNS side, so the useful conversation is about delivery, not about publishing.

A standard topic accepts 12,500,000 subscriptions, which covers this client's order of magnitude with room to spare. That number does not hold for a FIFO topic, which accepts 100 subscriptions per topic. Anyone considering a FIFO topic for the campaign to gain deduplication meets that limit before anything else.

Publish throughput per account and region varies: 30,000 messages per second in one group, 9,000 in another, 1,500 in a third, and 300 in the remaining supported regions, sa-east-1 included. For a weekly publish that is irrelevant, and I mention it because it is the number everyone looks up first and it is not the number that matters here.

What matters is the delivery rate per protocol, and those differ by orders of magnitude.

SMS has a delivery rate of 20 messages per second, for both promotional and transactional messages. For one million recipients that is 50,000 seconds of delivery, roughly 14 hours. A Saturday 9 AM campaign with a million SMS messages finishes in the early hours of Sunday. That calculation belongs in the conversation before anyone promises a delivery window to the marketing team.

Before volume there is the spend limit. The monthly SMS spending quota starts at USD 1.00 on a new account, which is a test budget. Production requires a Support case describing the use case, the opt-in process and the message template. The SNS pricing page also records that some countries require dedicated origination identities for sending to work at all, and that since November 1, 2024 SMS charges appear on the bill under AWS End User Messaging. None of those three gets resolved in the week of the campaign.

Email has a quota of 10 messages per second per email or email-json subscription, documented as a hard limit that cannot be raised. Watch the unit: per subscription, not per topic. It does not cap the aggregate fanout across millions of distinct subscriptions. The SNS quota documentation does not publish an aggregate email delivery rate, and I am not going to estimate one.

Message size: the default is 262,144 bytes, 256 KiB, and it can go up to 1 MiB through the MaximumMessageSize topic attribute. Each 64 KB chunk published counts as one request for billing.

How long SNS keeps retrying a delivery, by endpoint type

The rates above describe deliveries that work. The deliveries that fail follow a schedule SNS sets per protocol, and the alarm threshold in the template comes out of it.

SNS defines a delivery policy for each delivery protocol, and that policy governs how a server-side failure is retried. The spread between policies is wide. For an AWS managed endpoint, meaning Lambda or SQS, SNS retries up to 100,015 times over 23 days. For a customer managed endpoint, which is where SMTP, SMS and mobile push sit, it is 50 attempts over 6 hours. Same topic, same publish, and the window in which a broken receiver can come back and still get the message differs by orders of magnitude between the two. This campaign lives on the second number: with email and SMS subscriptions, a destination that is down for an afternoon has lost that week's message. HTTP/S is the one protocol where the policy is yours to write, through healthyRetryPolicy at the topic or the subscription level, and even there the total policy retry time cannot be greater than 3,600 seconds, which the documentation gives as a hard limit that cannot be raised.

That is also where the DeliveryFailureThreshold number comes from. numRetries covers the immediate, pre-backoff, backoff and post-backoff phases together, defaults to 3 and accepts up to 100, so an HTTP/S subscription can turn one undeliverable message into as many as 100 counts on NumberOfNotificationsFailed. Set the threshold above the retry count of the noisiest subscription on the topic, or the alarm reports a retry storm as a loss.

Retries are not the default outcome either. The delivery policy governs server-side failure, and for an HTTP/S endpoint the documentation is specific about which responses enter it: 5XX and 429 are retryable, and everything else is treated as a permanent failure. On the other protocols the division the documentation draws is between a server-side error, which the policy covers, and a client-side error, which it does not. A client-side error, which is the category covering a deleted endpoint or an endpoint policy that now refuses SNS, is attempted once and never retried.

The part that decides whether a failed delivery leaves anything behind is what happens after the policy runs out. SNS stops retrying and discards the message, unless a dead-letter queue is attached to the subscription. That queue is a redrive policy on AWS::SNS::Subscription, it attaches per subscription rather than per topic, and for a standard topic it points at a standard SQS queue in the same account and region. The dead letter queue in these templates is not that queue. It belongs to the trigger, and it catches a firing that never became a publish. After a successful publish the trigger side has nothing left to catch, so the recoverable evidence requirement is met for the firing and covers nothing past it: the fourth alarm counts the failed deliveries, and the messages behind that count are already gone.

Subscription dead-letter queues stayed out of both templates, for a reason of ownership rather than cost. The only subscription either stack creates is the test one, and a queue behind a test address records nothing anybody will read. The millions of production subscriptions are not created here and are not in scope for these stacks, so their redrive policies belong to whatever process creates subscriptions, sized per subscription class instead of one queue for the whole topic. Closing the delivery side is real work, and it is work on the subscription, not on the schedule.

At-least-once delivery and duplicate campaigns

EventBridge delivers with at-least-once semantics. Its own troubleshooting documentation has a section for the case of a rule running more than once in response to a single event, and describes it as rare rather than impossible. EventBridge Scheduler lists at-least-once delivery among the characteristics of the service.

For a marketing campaign, at-least-once means a risk of the subscriber getting the same email twice. That was exactly the complaint that opened this work, and replacing the human finger with a schedule lowers the probability without eliminating it.

How it materializes: one occurrence of the schedule produces two target invocations, two sns:Publish calls, two messages on the topic, two deliveries per subscription. Nothing on the path compares the two.

The mitigation depends on how strict the requirement is. The identifier in the payload, which is why executionId exists in the InputTransformer, makes the duplicate detectable by a consumer: retries of the same occurrence carry the same event id, and each weekly firing carries a new one. In the Scheduler template there is a difference worth knowing, and it sent me back to the documentation: aws.scheduler.execution-id is assigned per invocation attempt, not per occurrence. It does not work as a deduplication key. The key that is stable across attempts of the same occurrence is aws.scheduler.scheduled-time, and an aws.scheduler.attempt-number above 1 identifies a retry.

The uncomfortable part: detectable is not the same as deduplicated. With SNS delivering directly to email and SMS, there is no control point between the publish and the subscriber's inbox. To guarantee one delivery per occurrence, the design needs an intermediate step. The topic feeds a queue or a function, that consumer records the occurrence key conditionally, and only then triggers the send. That is a new component to maintain, which is what the minimum-operational-effort requirement was trying to avoid. The choice was made with open eyes: the client accepted a residual risk of rare duplicates in exchange for not having another service on the path, after I showed that the cause of both incidents was human and not infrastructural.

Metrics, alarms and what is opt-in

On the trigger side, in the AWS/Events namespace with dimension RuleName: TriggeredRules and Invocations for the success path, FailedInvocations for permanent failure, RetryInvocationAttempts to see instability that has not become failure yet, ThrottledRules for runs deferred by throttling, InvocationsSentToDlq and InvocationsFailedToBeSentToDlq for the dead letter path. Several of those are published only when non-zero, which changes how you build an alarm and a dashboard on them.

On the delivery side, in the AWS/SNS namespace: NumberOfMessagesPublished, NumberOfNotificationsDelivered, NumberOfNotificationsFailed and NumberOfNotificationsFilteredOut. The failure count behaves differently per protocol, as described above, so comparing that metric between topics with a different protocol mix leads to the wrong conclusion. SNS metrics are collected at 1-minute intervals, a topic counts as active for up to six hours after the last API call on it, and there is no charge for these metrics.

What a healthy week looks like on those metrics is worth writing down before building a dashboard, because the numbers are small and an empty graph is the expected state most of the time. Per Saturday, over the 5 minute period that contains the firing:

Namespace	Metric	Dimension	Healthy week
`AWS/Events`	`TriggeredRules`	`RuleName` = `weekly-campaigns-rule-saturday-trigger`	one data point, `Sum` of 1
`AWS/Events`	`Invocations`	same	one data point, `Sum` of 1
`AWS/Events`	`FailedInvocations`	same	no data point
`AWS/Events`	`InvocationsSentToDlq`	same	no data point
`AWS/SQS`	`ApproximateNumberOfMessagesVisible`	`QueueName` = `weekly-campaigns-rule-dlq`	`Maximum` of 0
`AWS/SNS`	`NumberOfMessagesPublished`	`TopicName` = `weekly-campaign-subscribers`	one data point, `Sum` of 1
`AWS/SNS`	`NumberOfNotificationsDelivered`	same	`Sum` in the order of the confirmed subscriptions
`AWS/SNS`	`NumberOfNotificationsFailed`	same	no data point, or a `Sum` below the alarm threshold

Those are the values this design predicts, not readings I took: no stack was deployed for this post. The useful part is the shape. A single-digit Sum on the trigger side and a seven-digit Sum on the delivery side belong to the same firing, and a Sum of 2 on Invocations inside one Saturday period is the duplicate the at-least-once section is about.

A caveat that applies to both sides: the EventBridge documentation states that metric delivery to CloudWatch is best effort, and that a data point may be delayed or not delivered. Metrics serve trend and alarm, not a complete accounting of operations.

SNS delivery status logging is opt-in, requires an IAM role, and is configured per protocol. It carries a limit that matters for this case: the documentation lists Amazon Data Firehose, Amazon SQS, AWS Lambda, HTTPS and platform application endpoint. Email is not on the list. For a topic whose subscriptions are mostly email, the per-subscription trace that exists is the aggregate metric, not a per-delivery log.

On Scheduler the metrics live in the AWS/Scheduler namespace: InvocationAttemptCount, TargetErrorCount, TargetErrorThrottledCount, InvocationThrottleCount, InvocationDroppedCount, and, when a dead letter queue is configured, InvocationsSentToDeadLetterCount and InvocationsFailedToBeSentToDeadLetterCount. They are kept for 15 months, which covers more than a year of campaigns.

What dominates the bill

Prices and quotas consulted on October 8, 2026, with the URLs in the references.

Scheduling costs close to nothing in both designs, for different reasons. The EventBridge pricing page does not list a charge for scheduled rule invocations on the default event bus, and event delivery to a service in the same account appears as USD 0.00 per million in the calculation examples. Scheduler has a free tier of 14 million invocations per month and charges USD 1.00 per million above that. A weekly campaign produces 52 invocations a year, five orders of magnitude below the free tier.

The EC2 instance is the only alternative with a relevant fixed cost, and the comparison does not need a price table to be settled: a resource billed by uptime, available 168 hours a week, to run a task of less than a second once a week. Any instance price multiplied by that utilization rate loses to a schedule that is not billed per invocation.

What dominates the bill is SNS delivery, not scheduling. The pricing structure of a standard topic has two components, API requests and deliveries, and the delivery cost depends on the protocol. Each 64 KB chunk published counts as a request and, except for SMS, each 64 KB chunk delivered counts as a delivery. Data leaving SNS for SQS or Lambda is billed as internet data transfer, regardless of the transfer happening inside the same region. SMS is priced per destination and varies by country, and it tends to be the line item that dominates the bill when it is in the mix. I am not reproducing unit prices, because they are not available in verifiable form on the pricing page without JavaScript rendering, and a wrong price in a post is worse than an absent one.

Least privilege, encryption and the account boundary

Who can publish to this topic is the whole security question, because a single sns:Publish call reaches every confirmed subscription with no approval step in between. There is no staging, no preview and no recall. The blast radius of that permission is the subscriber list.

The two designs put the permission in different places, and that changes what you audit. In the rule template the publish path lives in the topic resource policy, pinned to the EventBridge service principal, to the ARN of this rule, and to this account, and nothing else in that document grants anything. To answer who can publish, you read one statement. In the Scheduler template the topic has no resource policy at all: the publish is done by the execution role, pinned by its trust policy to this account and to this schedule group, with sns:Publish scoped to one topic ARN. To answer the same question there, you read one role. Both are narrow, and the operational difference is which artifact a reviewer has to open. What neither of them narrows is the identity policy side: any principal in the account whose IAM policy allows sns:Publish on this topic can publish, and finding those principals means reading identity policies, not this template.

Encryption at rest with a customer managed key is in the path because the publisher is a service principal, which is the requirement from the SNS documentation rather than a choice. It protects the message while SNS holds it and it gives the key policy as a second place where the publish can be denied. It stops at the point of delivery. The copy that lands in an inbox and the copy that lands on a phone are readable at the destination, and that is the nature of email and SMS, not a gap in the configuration. For a campaign the content is public by design, so this is not a finding here. For a notification carrying anything about an individual subscriber, the protocol choice is the control, and the encryption of the topic says nothing about it. The absence of delivery status logging for email, covered above, belongs to the same conversation: for most of these subscriptions there is no per-delivery record to audit.

The account boundary is where this design would need real work. Today the condition on the publish statement is !GetAtt CampaignRule.Arn, a reference resolved by CloudFormation inside the same stack. A rule in another account has an ARN this template cannot resolve, so the pin becomes a value you maintain by hand, and every account added to the campaign is a change to the topic policy rather than a change in the other account. The encrypted topic adds a second edit in the same move, because the key policy also has to admit the external caller, and the key policy is not written by the stack that owns the rule. On the Scheduler side the problem shifts rather than disappears: the execution role lives with the schedule, so a schedule in another account means a role in that account, and authorizing it is again a statement in this topic policy. The point to carry out of this section is that the single narrow statement that holds today holds because everything sits in one account, and that going cross-account is not a parameter.

Where this design stops serving

The design solves reliable scheduling for one message that is the same for everybody. It has no notion of a person.

When the requirement grows to include behavioral segmentation, content templates with per-recipient variables, granular opt-out by campaign type, or open and click reporting, none of that exists in SNS. A topic has subscriptions, and a subscription has a filter policy by attribute or by payload. You can separate audiences with subscription filters up to a point, and past "who opened last week's campaign" you cannot.

That is where the conversation used to turn to Amazon Pinpoint, and that recommendation no longer holds. AWS set the end of support for Pinpoint on October 30, 2026, two weeks after the date of this post. The service stopped accepting new customers on May 20, 2025, and after the cut-off date the console and the engagement features (endpoints, segments, campaigns, journeys and analytics) become inaccessible. The SMS, voice, push, OTP and phone number validation APIs are not affected: they were renamed to AWS End User Messaging in the third quarter of 2024 and continue to operate.

The official migration guide points to two paths. For engagement with segmentation, content templates and campaign orchestration, the proactive engagement solutions of Amazon Connect, including outbound campaigns and Customer Profiles. For email sending, Amazon Simple Email Service, with the Virtual Deliverability Manager replacing the deliverability dashboard. For mobile event collection and analytics, Amazon Kinesis. The guide also says when to stay on End User Messaging with SES instead of moving to Connect: a throughput requirement above the outbound campaign limits, and no need for a human agent or contact orchestration.

The honest limit of this post: it delivers the trigger, and the guarantee that the trigger leaves a trace. Lifecycle marketing is a different problem, with a different budget.

What the client ended up with

The Saturday campaign now fires on its own, at 09:00 in Sao Paulo rather than whenever someone got to it on Friday night, described in versioned YAML instead of a shared document with the topic ARN in bold. Over the following three months there was no duplicate firing, no missed firing and no firing at the wrong hour, and the dead letter queue stayed empty. The Friday night procedure stopped existing.

References

Setting a schedule pattern for scheduled rules (legacy) in Amazon EventBridge: the six cron fields, the wildcard table, the day-of-month and day-of-week restriction, the UTC+0 annotation on the examples, and the note that scheduled rules are a legacy feature.
How EventBridge retries delivering events: the default of 24 hours and up to 185 attempts with exponential backoff and jitter.
Using dead-letter queues to process undelivered events in EventBridge: standard queues only, same region, message attributes, the errors that skip retries, and the permission the console attaches and the API does not.
Amazon EventBridge input transformation: predefined variables, including aws.events.event.ingestion-time and aws.events.rule-name, and the quoting rules.
Amazon EventBridge events detail reference: the Scheduled Event detail type, source as aws.events, detail with no required fields, and the example event with the rule ARN in resources.
AWS service event metadata: the fields present in every event delivered to an event bus, id in UUID format generated per event, and detail that can be {}.
Amazon EventBridge events: Scheduled Event listed among the events EventBridge sends directly, with source aws.events and best effort delivery.
Creating a scheduled rule (legacy) in Amazon EventBridge: all scheduled events in UTC+0, one minute as the finest resolution, and the delay of several seconds between the trigger and the target running.
Monitoring Amazon EventBridge: the AWS/Events metrics, the RuleName dimension, publication only when non-zero, and best effort metric delivery.
Cross-service confused deputy prevention in Amazon EventBridge: aws:SourceArn has to be the rule ARN for a target of a rule on an event bus.
Using resource-based policies for Amazon EventBridge: the sns:Publish statement with principal events.amazonaws.com, shown without origin conditions.
AWS global condition context keys: aws:SourceAccount present only when a service principal calls the resource, and aws:PrincipalAccount present in every request.
Amazon SNS access policy language key concepts: aws:SourceAccount and aws:SourceOwner as the pair used when another AWS service publishes to the topic.
Example cases for Amazon SNS access control: the list of services that set SourceAccount when publishing to a topic, with Amazon EventBridge on it, the deprecation of aws:SourceOwner, and the note that a new topic is created private with no permissions.
Amazon SNS CreateTopic API reference: the action is idempotent, and a requester who already owns a topic with the given name gets that topic's ARN back without a new topic being created.
Amazon SNS GetTopicAttributes API reference: the error list for the operation, where a topic that does not exist is reported with the code NotFound and HTTP 404.
My rule ran more than once in response to one event: duplicate target invocation.
AWS::Events::Rule Target: the InputTransformer, RetryPolicy, DeadLetterConfig and Id properties.
AWS::SQS::Queue: MessageRetentionPeriod from 60 to 1,209,600 seconds with a default of 345,600, SqsManagedSseEnabled, and the two identifiers of a queue, with Ref returning the URL and GetAtt Arn the ARN.
AWS::CloudWatch::Alarm: TreatMissingData defaulting to missing, Ref returning the alarm name, and the alarm ARN in the form alarm:NAME.
Actions, resources, and condition keys for Amazon EventBridge Scheduler: the ARN format of a schedule, schedule/${GroupName}/${ScheduleName}, and of a schedule group, schedule-group/${GroupName}.
Find the alias name and alias ARN for a KMS key: the alias ARN format, with alias/NAME as the resource part.
AWS::Scheduler::Schedule: ScheduleExpressionTimezone, the required FlexibleTimeWindow, and the expression formats.
AWS::Scheduler::Schedule FlexibleTimeWindow: Mode as OFF or FLEXIBLE, MaximumWindowInMinutes from 1 to 1440.
AWS::Scheduler::Schedule Target: required RoleArn, plus Input, RetryPolicy and DeadLetterConfig.
Using templated targets in EventBridge Scheduler: Amazon SNS Publish as a templated target configured by topic ARN, and the requirement that the execution role carry the permission of the target operation.
Configuring a schedule's dead-letter queue in EventBridge Scheduler: the DLQ has to be a standard queue and a FIFO queue is not supported, delivery of the dead-letter event is authorized by sqs:SendMessage on the execution role permission policy, and the message attributes carried on the dead-letter event.
Adding context attributes in EventBridge Scheduler: scheduled-time, execution-id assigned per attempt, attempt-number, schedule-arn, and the documented example formats of each, with scheduled-time carrying a Z suffix and execution-id a short alphanumeric string rather than a UUID.
Confused deputy prevention in EventBridge Scheduler: trust policy with aws:SourceAccount and the aws:SourceArn of the schedule group.
Monitoring Amazon EventBridge Scheduler with Amazon CloudWatch: the AWS/Scheduler namespace, the single ScheduleGroup dimension, the metric list, and 15 month retention.
What is Amazon EventBridge Scheduler: at-least-once delivery among the service characteristics.
Amazon SNS key management: the customer managed key requirement for publishing by an AWS service, the kms:GenerateDataKey* and kms:Decrypt grant, the principal table listing events.amazonaws.com, the note that origin conditions are not supported on this path, and KMSAccessDeniedException.
Amazon SNS message delivery status: the protocols supported by delivery status logging.
Monitoring Amazon SNS topics using CloudWatch: the AWS/SNS metrics, the per-protocol behavior of NumberOfNotificationsFailed, the 1-minute interval, the six hour activity window, and the absence of charges.
Amazon SNS message delivery retries: one delivery policy per protocol and only HTTP/S customizable, the four phases, 100,015 attempts over 23 days for AWS managed endpoints against 50 attempts over 6 hours for customer managed ones, the 3,600 second hard limit on total HTTP/S retry time, numRetries defaulting to 3 with a maximum of 100, 5XX and 429 as the retryable errors on the HTTP/S delivery policy, and the message discarded on exhaustion unless the subscription has a dead-letter queue.
Amazon SNS dead-letter queues: the queue attaches to the subscription rather than the topic, through a redrive policy naming the queue ARN, in the same account and region, with no retry at all for client-side errors.
Amazon SNS endpoints and quotas: 12,500,000 subscriptions per standard topic and 100 per FIFO topic, the email delivery rate of 10 per second per subscription as a hard limit, 20 per second for promotional and transactional SMS, publish throughput per region, and message size from 256 KiB to 1 MiB.
Requesting increases to your monthly Amazon SNS SMS spending quota: the USD 1.00 default on a new account and the increase process.
Amazon SNS pricing: the standard topic pricing structure, the 64 KB chunk as the billing unit, transfer to SQS and Lambda billed as internet, dedicated origination identities for SMS in some countries, and SMS charges under AWS End User Messaging since November 1, 2024.
Amazon EventBridge pricing: the Scheduler free tier of 14 million invocations per month, USD 1.00 per million above it, and delivery to a service in the same account at USD 0.00 per million.
Amazon Pinpoint end of support: October 30, 2026, the end of new customers on May 20, 2025, the affected features, the APIs not affected under AWS End User Messaging, and the migration paths to Amazon Connect, SES and Kinesis.
Starting state machine executions in Step Functions: the list of ways an execution can start, none of them a scheduler internal to the service.
Using Amazon EventBridge Scheduler to start a Step Functions state machine execution: the official path for starting a state machine on a schedule.
AWS Well-Architected Framework, Operational Excellence pillar
AWS Well-Architected Framework, Reliability pillar
AWS Well-Architected Framework, Security pillar
AWS Well-Architected Framework, Cost Optimization pillar
AWS Well-Architected Framework, Sustainability pillar

Blocking Container deployments based on vulnerability scans

Carlos Filho — Wed, 07 Oct 2026 17:12:33 +0000

I got an AWS question and implemented it to make sure that the option is correct.

A company runs containerized microservices on ECS Fargate across multiple AWS accounts. The security team has four requirements: scan images before production deployment, block images with CRITICAL vulnerabilities automatically, do it without manual intervention, and centralize all findings in a single audit account.

Two solutions cover all four requirements.

Prerequisites

Check these before running terraform apply:

Amazon Inspector must be enabled per account
Inspector is not enabled by default. It must be activated in each member account before ECR Enhanced Scanning works. The recommended approach is enabling it through AWS Organizations so new accounts get it automatically.

Security Hub must be enabled in all accounts
Security Hub also requires explicit activation per account and per region. The aggregation feature requires Security Hub to be active in both the member accounts and the central security account.

The securityhub.tf file runs in a different account
The Security Hub aggregator resources must be applied to the central security account, not to each member account. The Terraform for that file requires a separate provider alias pointing to the aggregator account. This is covered in the securityhub.tf section below.

Terraform executor permissions
The IAM principal running Terraform needs at minimum:

inspector2:Enable
inspector2:AssociateMember
securityhub:EnableSecurityHub
securityhub:CreateFindingAggregator
securityhub:UpdateFindingAggregator
ecr:PutImageScanningConfiguration
ecr:DescribeImageScanFindings
events:PutRule
events:PutTargets
lambda:CreateFunction
lambda:AddPermission
iam:CreateRole
iam:AttachRolePolicy
iam:PutRolePolicy
sns:CreateTopic
sns:Subscribe

AdministratorAccess on each account covers all of these. Lock it down after the initial setup.

VPC not required
All resources in this solution run outside a VPC. Lambda, EventBridge, Inspector, and Security Hub are all regional services accessed via their public endpoints.

The problem

Blocking deployments automatically

A CI/CD pipeline that deploys on every image push will deploy vulnerable images unless something stops it. The gate needs to be automatic, not dependent on someone reviewing a dashboard. It also needs to be event-driven: when a scan finishes, the result should immediately trigger the enforcement logic.

The race condition

Inspector Enhanced Scanning does not complete instantly. Depending on image size and layer count, it can take two to five minutes after a push. If the CI/CD pipeline pushes an image and immediately triggers an ECS deployment, the deployment may start before Inspector finishes and before the Lambda gate has a chance to delete the image.

The safest mitigation is adding a wait step in the pipeline after the push, before the deploy step. A simple approach is polling the Inspector findings API until the scan status returns COMPLETE or a timeout is reached. If the image has been deleted by the gate Lambda during that window, the deploy step will fail when it tries to pull the image. If no CRITICAL findings exist, the image remains in ECR and the deploy proceeds normally.

A secondary mitigation is separating the ECR repository from the deploy step by requiring the deploy step to always reference an image by digest rather than by tag. This prevents a race where a new push overwrites a tag that was already scanned clean.

Centralizing findings across accounts

In a multi-account setup, each account has its own Inspector findings. The security team needs a single place to review all findings across all accounts without logging into each one individually.

Continuous scanning behavior

Enhanced Scanning is not a one-time check on push. Inspector re-evaluates images continuously as new CVEs are published. An image that passes the gate today may be flagged tomorrow if a new CRITICAL vulnerability is discovered in one of its packages. The EventBridge rule will fire again for the new finding, and the Lambda gate will delete the image from ECR even if it is already deployed.

This means deployed containers are not automatically stopped. The gate only prevents new pulls of the vulnerable image. If your ECS task uses imagePullPolicy: always equivalent behavior or if tasks restart, they will fail to pull the deleted image and the task will not start. This is the intended behavior: the image is gone, and a new clean build is required.

The solutions

ECR Enhanced Scanning with Inspector + EventBridge gate

ECR Enhanced Scanning uses Amazon Inspector instead of the built-in basic scanner. Enhanced Scanning checks OS packages and application dependencies continuously, not just on push. It cross-references findings against multiple vulnerability databases (NVD, vendor advisories) and assigns severity scores more accurately than Basic Scanning.

When Inspector finishes scanning an image, it emits an event to EventBridge. The event contains the image digest, repository, and a summary of findings by severity. An EventBridge rule filters for events where CRITICAL findings are present and routes them to a Lambda function.

The Lambda function deletes the image from ECR so it cannot be pulled, then notifies the pipeline via SNS so the deployment step fails with a clear reason.

The pipeline does not need a separate approval gate or polling loop. It pushes the image, the scan runs, and if CRITICAL vulnerabilities are found the image is deleted before the deployment step can reference it.

# inspector.tf

resource "aws_inspector2_enabler" "ecr" {
  account_ids    = [data.aws_caller_identity.current.account_id]
  resource_types = ["ECR"]
}

data "aws_caller_identity" "current" {}

With Inspector enabled for ECR, all repositories in the account get continuous scanning automatically. No per-repository configuration is needed.

# ecr.tf

resource "aws_ecr_repository" "app" {
  for_each = toset(var.ecr_repository_names)

  name                 = each.key
  image_tag_mutability = "IMMUTABLE"

  image_scanning_configuration {
    scan_on_push = true
  }

  encryption_configuration {
    encryption_type = "AES256"
  }

  tags = {
    Name = each.key
  }
}

# Clean up untagged images left behind after the gate Lambda deletes a vulnerable image.
# Without this, deleted images accumulate as untagged layers and incur storage cost.
resource "aws_ecr_lifecycle_policy" "app" {
  for_each   = aws_ecr_repository.app
  repository = each.value.name

  policy = jsonencode({
    rules = [
      {
        rulePriority = 1
        description  = "Remove untagged images immediately"
        selection = {
          tagStatus   = "untagged"
          countType   = "sinceImagePushed"
          countUnit   = "days"
          countNumber = 1
        }
        action = {
          type = "expire"
        }
      },
      {
        rulePriority = 2
        description  = "Keep only the last 20 tagged images per repository"
        selection = {
          tagStatus     = "tagged"
          tagPrefixList = ["v"]
          countType     = "imageCountMoreThan"
          countNumber   = 20
        }
        action = {
          type = "expire"
        }
      }
    ]
  })
}

image_tag_mutability = "IMMUTABLE" prevents overwriting an existing tag. Combined with the scan gate, this means a tag that passed scanning cannot be replaced with a vulnerable image under the same tag.

# iam.tf

resource "aws_iam_role" "scan_gate_lambda" {
  name = "ECRScanGateLambdaRole"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect    = "Allow"
      Principal = { Service = "lambda.amazonaws.com" }
      Action    = "sts:AssumeRole"
    }]
  })
}

resource "aws_iam_role_policy" "scan_gate_lambda" {
  name = "ECRScanGateLambdaPolicy"
  role = aws_iam_role.scan_gate_lambda.id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Sid    = "ECRDeleteImage"
        Effect = "Allow"
        Action = [
          "ecr:BatchDeleteImage",
          "ecr:DescribeImages",
          "ecr:DescribeImageScanFindings"
        ]
        Resource = [for repo in aws_ecr_repository.app : repo.arn]
      },
      {
        Sid      = "SNSPublish"
        Effect   = "Allow"
        Action   = "sns:Publish"
        Resource = aws_sns_topic.scan_alerts.arn
      },
      {
        Sid    = "Logs"
        Effect = "Allow"
        Action = [
          "logs:CreateLogGroup",
          "logs:CreateLogStream",
          "logs:PutLogEvents"
        ]
        Resource = "arn:aws:logs:*:*:*"
      }
    ]
  })
}

# lambda.tf

resource "aws_lambda_function" "scan_gate" {
  function_name = "ecr-scan-gate"
  role          = aws_iam_role.scan_gate_lambda.arn
  handler       = "index.handler"
  runtime       = "python3.12"
  timeout       = 30

  filename         = data.archive_file.scan_gate.output_path
  source_code_hash = data.archive_file.scan_gate.output_base64sha256

  environment {
    variables = {
      SNS_TOPIC_ARN = aws_sns_topic.scan_alerts.arn
    }
  }
}

data "archive_file" "scan_gate" {
  type        = "zip"
  output_path = "${path.module}/scan_gate.zip"

  source {
    content  = <<-PYTHON
import boto3
import os
import json

ecr = boto3.client('ecr')
sns = boto3.client('sns')

def handler(event, context):
    detail = event.get('detail', {})
    repository = detail.get('repository-name')
    tag = detail.get('image-tags', ['untagged'])[0]
    digest = detail.get('image-digest')
    findings_summary = detail.get('finding-severity-counts', {})
    critical_count = findings_summary.get('CRITICAL', 0)

    print(f"Repository: {repository}, Tag: {tag}, CRITICAL: {critical_count}")

    if critical_count > 0:
        print(f"Deleting image {repository}:{tag} ({digest}) - {critical_count} CRITICAL findings")

        ecr.batch_delete_image(
            repositoryName=repository,
            imageIds=[{'imageDigest': digest}]
        )

        sns.publish(
            TopicArn=os.environ['SNS_TOPIC_ARN'],
            Subject=f"Deployment blocked: {repository}:{tag}",
            Message=json.dumps({
                'repository': repository,
                'tag': tag,
                'digest': digest,
                'critical_findings': critical_count,
                'action': 'image deleted from ECR',
                'reason': 'CRITICAL vulnerabilities found during Inspector scan'
            }, indent=2)
        )

    return {'statusCode': 200}
    PYTHON
    filename = "index.py"
  }
}

resource "aws_lambda_permission" "eventbridge" {
  statement_id  = "AllowEventBridgeInvocation"
  action        = "lambda:InvokeFunction"
  function_name = aws_lambda_function.scan_gate.function_name
  principal     = "events.amazonaws.com"
  source_arn    = aws_cloudwatch_event_rule.ecr_scan_critical.arn
}

# eventbridge.tf

resource "aws_cloudwatch_event_rule" "ecr_scan_critical" {
  name        = "ecr-inspector-scan-complete"
  description = "Triggers when Inspector finds a CRITICAL vulnerability in an ECR image"

  event_pattern = jsonencode({
    source      = ["aws.inspector2"]
    detail-type = ["Inspector2 Finding"]
    detail = {
      resources = {
        type = ["AWS_ECR_CONTAINER_IMAGE"]
      }
      severity = ["CRITICAL"]
    }
  })
}

resource "aws_cloudwatch_event_target" "scan_gate_lambda" {
  rule      = aws_cloudwatch_event_rule.ecr_scan_critical.name
  target_id = "ScanGateLambda"
  arn       = aws_lambda_function.scan_gate.arn
}

# sns.tf

resource "aws_sns_topic" "scan_alerts" {
  name = "ecr-scan-critical-alerts"
}

resource "aws_sns_topic_subscription" "email" {
  topic_arn = aws_sns_topic.scan_alerts.arn
  protocol  = "email"
  endpoint  = var.pipeline_notification_email
}

Inspector findings to Security Hub with cross-account aggregation

Inspector publishes all ECR findings directly to Security Hub with no additional configuration beyond enabling both services. Security Hub normalizes findings into the ASFF (Amazon Security Finding Format), making them queryable and filterable in a consistent way regardless of which account or scanner produced them.

Security Hub's finding aggregation feature designates one account as the aggregator. All member accounts linked through Organizations forward their findings to the aggregator account automatically. The security team reviews findings in one place, sets up alerting once, and writes queries once.

No custom code. No cross-account Lambda invocations. No S3 buckets collecting logs from each account.

The securityhub.tf below must be applied to the central security account, not to member accounts. Use a separate Terraform provider alias pointing to that account.

# securityhub.tf
# Apply this with the security account provider alias, not the default provider.
# Example provider configuration in main.tf:
#
# provider "aws" {
#   alias  = "security"
#   region = var.aws_region
#   assume_role {
#     role_arn = "arn:aws:iam::${var.security_hub_aggregator_account_id}:role/TerraformRole"
#   }
# }

resource "aws_securityhub_account" "aggregator" {
  provider = aws.security
}

resource "aws_securityhub_finding_aggregator" "main" {
  provider     = aws.security
  linking_mode = "ALL_REGIONS"

  depends_on = [aws_securityhub_account.aggregator]
}

resource "aws_securityhub_member" "members" {
  provider = aws.security
  for_each = toset(var.member_account_ids)

  account_id = each.key
  invite     = true

  depends_on = [aws_securityhub_account.aggregator]
}

resource "aws_securityhub_product_subscription" "inspector" {
  provider    = aws.security
  product_arn = "arn:aws:securityhub:${var.aws_region}::product/aws/inspector"

  depends_on = [aws_securityhub_account.aggregator]
}

Full variable list

# variables.tf

variable "aws_region" {
  description = "AWS region"
  type        = string
  default     = "us-east-1"
}

variable "ecr_repository_names" {
  description = "List of ECR repository names to protect"
  type        = list(string)
}

variable "pipeline_notification_email" {
  description = "Email address that receives notifications when a deployment is blocked"
  type        = string
}

variable "security_hub_aggregator_account_id" {
  description = "AWS account ID of the central security account that aggregates findings"
  type        = string
}

variable "member_account_ids" {
  description = "List of member account IDs to link to the Security Hub aggregator"
  type        = list(string)
}

Outputs

# outputs.tf

output "scan_gate_lambda_arn" {
  description = "ARN of the Lambda function that enforces the scan gate"
  value       = aws_lambda_function.scan_gate.arn
}

output "scan_alerts_topic_arn" {
  description = "SNS topic ARN that receives blocked deployment notifications"
  value       = aws_sns_topic.scan_alerts.arn
}

output "ecr_repository_urls" {
  description = "ECR repository URLs for the CI/CD pipeline"
  value       = { for k, v in aws_ecr_repository.app : k => v.repository_url }
}

Validating the deployment

Check 1: Inspector is enabled for ECR

aws inspector2 list-coverage \
  --filter-criteria '{"resourceType":[{"comparison":"EQUALS","value":"AWS_ECR_CONTAINER_IMAGE"}]}' \
  --query 'coveredResources[0].scanStatus.statusCode'
# Expected: "ACTIVE"

Check 2: EventBridge rule is active

aws events describe-rule \
  --name ecr-inspector-scan-complete \
  --query 'State'
# Expected: "ENABLED"

Check 3: trigger the gate with a known vulnerable image

Push an image with a known CRITICAL vulnerability to one of the protected repositories. Inspector will scan it within a few minutes and the EventBridge rule will fire. Confirm the image was deleted:

aws ecr list-images \
  --repository-name <your-repo-name> \
  --query 'imageIds[*]'

The image should no longer appear. Check the SNS email for the blocked deployment notification.

Check 4: Security Hub is receiving Inspector findings

Run this from the aggregator account:

aws securityhub get-findings \
  --filters '{"ProductName":[{"Value":"Inspector","Comparison":"EQUALS"}]}' \
  --query 'Findings[0].{Title:Title,Severity:Severity.Label,Account:AwsAccountId}' \
  --region <aggregator-region>

Findings from all member accounts should appear here.

Why the other options do not work?

Option 1: Basic Scanning with CodePipeline approval gate

Basic Scanning only runs on push and checks OS packages against a single vulnerability feed. It misses application-layer vulnerabilities that Inspector catches. The CodePipeline approval gate also requires per-pipeline integration: every pipeline that deploys from ECR needs its own gate, and each gate needs to know which image it is checking. This scales poorly across multiple repositories and pipelines.

Option 2: CloudWatch metric filter on ECR scan events

ECR scan result events go to EventBridge natively. Routing them through CloudWatch Logs first adds a step with no benefit. You lose the structured event payload, need a metric filter pattern to extract severity, and then need to reconnect back to a Lambda anyway. EventBridge is the direct path.

Option 3: AWS Config with ecr-image-scan-on-push rule

The Config managed rule checks whether scan-on-push is enabled for a repository. It evaluates repository configuration, not scan results. It has no visibility into whether a scan found CRITICAL vulnerabilities. Remediating non-compliance means enabling scan-on-push on repositories that have it disabled, not blocking images based on findings.

Architecture

The two flows run in parallel. Inspector feeds both the blocking gate and the centralized audit trail at the same time.

CI/CD pipeline
    |
    v
docker push --> ECR repository (IMMUTABLE tags)
                    |
              Inspector scans image (2-5 min)
                    |
          +---------+---------+
          |                   |
          v                   v
   EventBridge rule     Security Hub
   (CRITICAL finding)   (all findings, all accounts)
          |                   |
          v                   v
   Lambda gate          Aggregator account
          |             Single audit view
   CRITICAL > 0?
          |
     Yes  |  No
     |         |
     v         v
  Delete    Image stays
  image     Pipeline deploys
  from ECR  normally
     |
  SNS notification
  Pipeline fails
  (image not found)

Inspector also re-evaluates images continuously.
New CVE published --> new EventBridge event --> gate fires again.
Deployed containers are not stopped automatically,
but the image is removed from ECR so no new tasks can pull it.

Cost to keep in mind

Amazon Inspector charges per container image scanned per month. The first 2,500 images per account per month are included in the free tier during the first 30 days. After that it charges per image. Security Hub charges per finding ingested, with a free tier of 10,000 findings per month per account for the first 12 months.

For a multi-account environment with active CI/CD pipelines, Inspector and Security Hub costs scale with the number of images pushed. The ECR lifecycle policy prevents untagged layer accumulation from inflating storage costs when the gate deletes images.

RDS High Availability and credential rotation without downtime

Carlos Filho — Tue, 18 Aug 2026 18:54:22 +0000

I got an AWS question and implemented it to make sure that the option is correct.

A critical financial application runs on RDS for PostgreSQL. The requirements are tight: 1-second RPO, 60-second RTO, and database credentials rotated every 30 days without taking the application offline.

Two independent problems. Two independent solutions.

Prerequisites

Check these before running terraform apply:

RDS Proxy availability
RDS Proxy is not available on all instance types. It requires instances with at least 2 vCPUs. db.t3.micro is not supported. db.t3.medium and above work.

Terraform executor permissions
The IAM principal running Terraform needs, at minimum:

rds:CreateDBInstance
rds:CreateDBProxy
rds:CreateDBProxyTargetGroup
rds:RegisterDBProxyTargets
rds:ModifyDBInstance
iam:CreateRole
iam:AttachRolePolicy
iam:PutRolePolicy
iam:PassRole
secretsmanager:CreateSecret
secretsmanager:PutSecretValue
secretsmanager:RotateSecret
lambda:CreateFunction
lambda:AddPermission
ec2:CreateSecurityGroup
ec2:AuthorizeSecurityGroupIngress
ec2:CreateDBSubnetGroup

AdministratorAccess on the account covers all of these. Lock it down after the initial setup.

VPC requirements
RDS Proxy runs inside your VPC. You need at least two private subnets in different Availability Zones. The rotation Lambda also runs inside the VPC so it can reach the RDS instance directly during the credential update step.

The problem

Database failure recovery

RPO of 1 second means almost no data loss is acceptable. RTO of 60 seconds means the application must resume within a minute of a failure. A standard single-instance RDS setup fails both requirements: there is no automatic failover, and restoring from a backup takes far longer than 60 seconds.

Credential rotation

Rotating credentials on a schedule sounds simple until you factor in application downtime. If you update a password and the application still holds connections authenticated with the old one, those connections fail. The rotation mechanism needs to handle the transition without the application noticing.

The solutions

Multi-AZ + RDS Proxy

RDS Multi-AZ keeps a synchronous standby replica in a second Availability Zone. Every write to the primary is committed to the standby before the transaction is acknowledged. This gives you RPO close to zero, typically under 1 second for PostgreSQL.

When the primary fails, RDS automatically promotes the standby. The DNS record for the cluster endpoint is updated to point to the new primary. Failover typically completes in 60 seconds or less, which meets the RTO requirement.

The problem is what happens to application connections during that DNS update. Existing database connections are dropped. If your application has a connection pool, it will spend time re-establishing connections and waiting for DNS propagation. Under load, this causes errors.

RDS Proxy sits between the application and the database. The application connects to the Proxy endpoint, not the RDS endpoint directly. The Proxy maintains a warm pool of connections to the database. During a Multi-AZ failover, the Proxy detects the new primary and reroutes connections internally. The application connection to the Proxy stays alive. From the application's perspective, the failover is nearly invisible.

Without the Proxy, Multi-AZ alone may technically meet 60-second RTO on paper but cause application-level errors during the DNS transition. With the Proxy, the application keeps working through the failover.

Secrets Manager with automatic rotation

Secrets Manager stores the database credentials and handles rotation through a managed Lambda function. AWS provides a built-in rotation Lambda for RDS PostgreSQL that handles the full rotation cycle: generate a new password, update it in RDS, update the secret, verify the new credentials work.

The rotation Lambda follows a four-step process using staging labels (AWSPENDING, AWSCURRENT, AWSPREVIOUS). The new password is set in the database before it becomes AWSCURRENT. This means both the old and new passwords are valid during the transition window. Applications retrieving credentials from Secrets Manager at runtime always get the current valid credentials without any downtime.

RDS Proxy reads credentials from Secrets Manager directly. The application connects to the Proxy endpoint and never touches credentials directly. When Secrets Manager rotates the password, the Proxy picks up the new credentials on its next refresh cycle. The application connection to the Proxy stays alive throughout.

Terraform

File structure

├── variables.tf
├── networking.tf
├── iam.tf
├── rds.tf
├── secrets.tf
└── outputs.tf

`variables.tf`

variable "aws_region" {
  description = "AWS region to deploy resources"
  type        = string
  default     = "us-east-1"
}

variable "vpc_id" {
  description = "ID of the VPC where RDS and RDS Proxy will run"
  type        = string
}

variable "private_subnet_ids" {
  description = "List of private subnet IDs in at least two AZs"
  type        = list(string)
}

variable "app_security_group_id" {
  description = "Security group ID of the application servers that connect to RDS Proxy"
  type        = string
}

variable "db_password" {
  description = "Initial database password (Secrets Manager takes over after first rotation)"
  type        = string
  sensitive   = true
}

`networking.tf`: security groups

# Security group for the RDS instance
# Accepts connections from RDS Proxy and the rotation Lambda only
resource "aws_security_group" "rds" {
  name        = "financial-app-rds"
  description = "Allow PostgreSQL access from RDS Proxy and rotation Lambda"
  vpc_id      = var.vpc_id

  ingress {
    description     = "PostgreSQL from RDS Proxy"
    from_port       = 5432
    to_port         = 5432
    protocol        = "tcp"
    security_groups = [aws_security_group.rds_proxy.id, aws_security_group.rotation_lambda.id]
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

# Security group for RDS Proxy
# Accepts connections from the application tier
resource "aws_security_group" "rds_proxy" {
  name        = "financial-app-rds-proxy"
  description = "Allow PostgreSQL access from application servers"
  vpc_id      = var.vpc_id

  ingress {
    description     = "PostgreSQL from application"
    from_port       = 5432
    to_port         = 5432
    protocol        = "tcp"
    security_groups = [var.app_security_group_id]
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

# Security group for the rotation Lambda
# Needs outbound access to RDS and to the Secrets Manager VPC endpoint (or internet)
resource "aws_security_group" "rotation_lambda" {
  name        = "financial-app-rotation-lambda"
  description = "Allow rotation Lambda to reach RDS and Secrets Manager"
  vpc_id      = var.vpc_id

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

# DB subnet group: RDS requires subnets in at least two AZs
resource "aws_db_subnet_group" "main" {
  name       = "financial-app-db-subnet-group"
  subnet_ids = var.private_subnet_ids

  tags = {
    Name = "financial-app-db-subnet-group"
  }
}

`iam.tf`: roles for RDS Proxy and rotation Lambda

# Role for RDS Proxy to read credentials from Secrets Manager
resource "aws_iam_role" "rds_proxy_role" {
  name = "RDSProxySecretsManagerRole"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect    = "Allow"
      Principal = { Service = "rds.amazonaws.com" }
      Action    = "sts:AssumeRole"
    }]
  })
}

resource "aws_iam_role_policy" "rds_proxy_secrets" {
  name = "RDSProxySecretsManagerPolicy"
  role = aws_iam_role.rds_proxy_role.id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect = "Allow"
      Action = [
        "secretsmanager:GetSecretValue",
        "secretsmanager:DescribeSecret"
      ]
      Resource = aws_secretsmanager_secret.db_credentials.arn
    }]
  })
}

`rds.tf`: RDS instance and Proxy

resource "aws_db_instance" "primary" {
  identifier              = "financial-app-db"
  engine                  = "postgres"
  engine_version          = "15.4"
  instance_class          = "db.t3.medium"
  allocated_storage       = 100
  db_name                 = "financialdb"
  username                = "dbadmin"
  password                = var.db_password
  multi_az                = true
  storage_encrypted       = true
  backup_retention_period = 7
  skip_final_snapshot     = false

  vpc_security_group_ids = [aws_security_group.rds.id]
  db_subnet_group_name   = aws_db_subnet_group.main.name

  tags = {
    Name = "financial-app-db"
  }
}

resource "aws_db_proxy" "main" {
  name                   = "financial-app-proxy"
  debug_logging          = false
  engine_family          = "POSTGRESQL"
  idle_client_timeout    = 1800
  require_tls            = true
  role_arn               = aws_iam_role.rds_proxy_role.arn
  vpc_security_group_ids = [aws_security_group.rds_proxy.id]
  vpc_subnet_ids         = var.private_subnet_ids

  auth {
    auth_scheme = "SECRETS"
    iam_auth    = "DISABLED"
    secret_arn  = aws_secretsmanager_secret.db_credentials.arn
  }

  tags = {
    Name = "financial-app-proxy"
  }
}

resource "aws_db_proxy_default_target_group" "main" {
  db_proxy_name = aws_db_proxy.main.name

  connection_pool_config {
    connection_borrow_timeout    = 120
    max_connections_percent      = 100
    max_idle_connections_percent = 50
  }
}

resource "aws_db_proxy_target" "main" {
  db_instance_identifier = aws_db_instance.primary.identifier
  db_proxy_name          = aws_db_proxy.main.name
  target_group_name      = aws_db_proxy_default_target_group.main.name
}

The application connects to aws_db_proxy.main.endpoint, not to the RDS instance endpoint directly.

`secrets.tf`: Secrets Manager and rotation

The rotation Lambda comes from the AWS Serverless Application Repository. AWS maintains it. You do not write or host any rotation code.

resource "aws_secretsmanager_secret" "db_credentials" {
  name                    = "financial-app/db-credentials"
  recovery_window_in_days = 7

  tags = {
    Name = "financial-app-db-credentials"
  }
}

resource "aws_secretsmanager_secret_version" "db_credentials" {
  secret_id = aws_secretsmanager_secret.db_credentials.id

  secret_string = jsonencode({
    username             = "dbadmin"
    password             = var.db_password
    engine               = "postgres"
    host                 = aws_db_instance.primary.address
    port                 = 5432
    dbname               = "financialdb"
    dbInstanceIdentifier = aws_db_instance.primary.identifier
  })
}

# Deploy the AWS-managed rotation Lambda from the Serverless Application Repository
resource "aws_serverlessapplicationrepository_cloudformation_stack" "rotation_lambda" {
  name             = "SecretsManagerRDSPostgreSQLRotationSingleUser"
  application_id   = "arn:aws:serverlessrepo:us-east-1:297356227824:applications/SecretsManagerRDSPostgreSQLRotationSingleUser"
  semantic_version = "1.1.387"

  capabilities = [
    "CAPABILITY_IAM",
    "CAPABILITY_RESOURCE_POLICY"
  ]

  parameters = {
    endpoint            = "https://secretsmanager.${var.aws_region}.amazonaws.com"
    functionName        = "SecretsManagerRDSPostgreSQLRotation"
    vpcSubnetIds        = join(",", var.private_subnet_ids)
    vpcSecurityGroupIds = aws_security_group.rotation_lambda.id
  }
}

data "aws_lambda_function" "rotation" {
  function_name = "SecretsManagerRDSPostgreSQLRotation"
  depends_on    = [aws_serverlessapplicationrepository_cloudformation_stack.rotation_lambda]
}

resource "aws_lambda_permission" "secrets_manager" {
  statement_id  = "AllowSecretsManagerInvocation"
  action        = "lambda:InvokeFunction"
  function_name = data.aws_lambda_function.rotation.function_name
  principal     = "secretsmanager.amazonaws.com"
}

resource "aws_secretsmanager_secret_rotation" "db_credentials" {
  secret_id           = aws_secretsmanager_secret.db_credentials.id
  rotation_lambda_arn = data.aws_lambda_function.rotation.arn

  rotation_rules {
    automatically_after_days = 30
  }

  depends_on = [aws_lambda_permission.secrets_manager]
}

`outputs.tf`

output "proxy_endpoint" {
  description = "Connect the application to this endpoint, not to the RDS instance directly"
  value       = aws_db_proxy.main.endpoint
}

output "secret_arn" {
  description = "ARN of the Secrets Manager secret storing database credentials"
  value       = aws_secretsmanager_secret.db_credentials.arn
}

output "rds_instance_id" {
  description = "RDS instance identifier"
  value       = aws_db_instance.primary.identifier
}

Validating the deployment

Check 1: Multi-AZ is active

aws rds describe-db-instances \
  --db-instance-identifier financial-app-db \
  --query 'DBInstances[0].MultiAZ'
# Expected: true

Check 2: RDS Proxy is available and connected to the instance

aws rds describe-db-proxies \
  --db-proxy-name financial-app-proxy \
  --query 'DBProxies[0].Status'
# Expected: "available"

aws rds describe-db-proxy-targets \
  --db-proxy-name financial-app-proxy \
  --query 'Targets[*].{Endpoint:Endpoint,State:TargetHealth.State}'
# Expected: State "AVAILABLE"

Check 3: rotation is configured and working

aws secretsmanager describe-secret \
  --secret-id financial-app/db-credentials \
  --query '{RotationEnabled:RotationEnabled,LastRotatedDate:LastRotatedDate,NextRotationDate:NextRotationDate}'

To trigger a manual rotation immediately and confirm the Lambda executes without errors:

aws secretsmanager rotate-secret \
  --secret-id financial-app/db-credentials

Then check the rotation status:

aws secretsmanager describe-secret \
  --secret-id financial-app/db-credentials \
  --query 'RotationRules'

Check 4: simulate a failover

RDS provides a reboot-with-failover option that forces the standby to become primary. This lets you observe the actual failover behavior before a real incident:

aws rds reboot-db-instance \
  --db-instance-identifier financial-app-db \
  --force-failover

Monitor the event log during the failover:

aws rds describe-events \
  --source-identifier financial-app-db \
  --source-type db-instance \
  --duration 10 \
  --query 'Events[*].{Time:Date,Message:Message}'

The application should continue serving requests through the Proxy during this window. Check your application logs for connection errors. If the Proxy is working correctly, you should see none.

Why the other options do not work?

Option: automated backups with PITR

PITR restores to a point in time by replaying transaction logs onto a snapshot. This creates a new RDS instance. Provisioning a new instance, restoring data, and updating connection strings takes 15 to 30 minutes at minimum. The 60-second RTO is not achievable with this approach.

Option: Parameter Store with EventBridge rotation

Parameter Store does not have native RDS credential rotation. You write the rotation logic yourself in Lambda, manage the staging window yourself, and handle the transition between old and new credentials yourself. This is exactly what Secrets Manager already does, with managed Lambda functions and staging label support built in. Option D reimplements a subset of Secrets Manager without the reliability guarantees.

Option: read replica with manual promotion

A read replica provides an asynchronous copy of the data. Promotion requires manual steps: stopping replication, promoting the replica, updating the application connection string. The process takes several minutes and requires human intervention. Neither the RTO nor the automation requirement is met.

What the architecture looks like end to end

Application
    |
    v
RDS Proxy endpoint
    |
    |-- Normal operation --> RDS Primary (AZ-a)
    |                             |
    |                         Synchronous replication
    |                             |
    |                        RDS Standby (AZ-b)
    |
    |-- During failover --> Proxy detects new primary
                            Proxy reroutes internally
                            Application connection stays alive

Secrets Manager (30-day rotation schedule)
    |
    v
Rotation Lambda (inside VPC)
    |-- Updates password in RDS
    |-- Updates AWSCURRENT in Secrets Manager
    |
RDS Proxy reads new credentials on next refresh
Application never sees the credential change

Cost to keep in mind

Multi-AZ doubles the instance cost because the standby runs continuously. RDS Proxy charges per vCPU of the underlying RDS instance per hour. Secrets Manager charges $0.40 per secret per month plus $0.05 per 10,000 API calls.

For a financial application with a 1-second RPO and 60-second RTO requirement, the alternative to this setup is manual failover, potential data loss, and credential management code you have to maintain and test yourself.

Enforcing SSE-KMS on S3 buckets across all AWS accounts

Carlos Filho — Thu, 16 Jul 2026 15:05:08 +0000

I got an AWS question and implemented it to make sure that the option is correct.

You have 30 AWS accounts under Organizations. A developer in one of them creates an S3 bucket, skips encryption, and now you have a compliance gap. By the time your next audit runs, there are three more buckets in the same state.

This post covers how to close that gap permanently: stopping unencrypted buckets before they are created, and fixing the ones that already exist.

Prerequisites

Check these before running terraform apply:

AWS Organizations

Organizations are enabled, and SCPs are active. Go to the management account → AWS Organizations → Policies → Service Control Policies and confirm it shows "Enabled". If SCPs are not enabled, the policy in this post will be created but will have no effect.

Terraform executor permissions
The IAM principal running Terraform needs, at minimum:

organizations:CreatePolicy
organizations:AttachPolicy
organizations:DescribePolicy
iam:CreateRole
iam:AttachRolePolicy
iam:PassRole
config:PutConfigRule
config:PutRemediationConfigurations
config:PutConfigurationRecorder
config:PutDeliveryChannel
config:StartConfigurationRecorder
cloudformation:CreateStackSet
cloudformation:CreateStackInstances
cloudformation:DescribeStackSet

AdministratorAccess on the management account covers all of these. Lock it down after the initial setup.

Config delivery bucket
AWS Config needs an S3 bucket to store configuration snapshots and history. The bucket must exist before you run terraform apply, and it needs a specific bucket policy that allows Config to write to it.

# config_bucket.tf - create this before the rest
resource "aws_s3_bucket" "config_delivery" {
  bucket        = "config-delivery-${data.aws_caller_identity.current.account_id}"
  force_destroy = false
}

resource "aws_s3_bucket_policy" "config_delivery" {
  bucket = aws_s3_bucket.config_delivery.id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Sid    = "AWSConfigBucketPermissionsCheck"
        Effect = "Allow"
        Principal = { Service = "config.amazonaws.com" }
        Action   = "s3:GetBucketAcl"
        Resource = aws_s3_bucket.config_delivery.arn
      },
      {
        Sid    = "AWSConfigBucketDelivery"
        Effect = "Allow"
        Principal = { Service = "config.amazonaws.com" }
        Action   = "s3:PutObject"
        Resource = "${aws_s3_bucket.config_delivery.arn}/AWSLogs/*/Config/*"
        Condition = {
          StringEquals = {
            "s3:x-amz-acl" = "bucket-owner-full-control"
          }
        }
      }
    ]
  })
}

data "aws_caller_identity" "current" {}

Without this exact bucket policy, Config will fail to start the recorder, and you will see InsufficientDeliveryPolicyException errors.

How does the solution work?

SCP: the preventive layer

A Service Control Policy attached to the root OU acts as a guardrail across every account in the organization. It does not grant permissions. It restricts what IAM can allow. Even an account admin cannot override it.

The policy denies s3:CreateBucket unless the request includes the SSE-KMS header. The bucket never gets created if encryption is not specified.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DenyS3CreateBucketWithoutSSEKMS",
      "Effect": "Deny",
      "Action": "s3:CreateBucket",
      "Resource": "*",
      "Condition": {
        "StringNotEqualsIfExists": {
          "s3:x-amz-server-side-encryption": "aws:kms"
        },
        "Null": {
          "s3:x-amz-server-side-encryption": "true"
        }
      }
    }
  ]
}

The StringNotEqualsIfExists + Null combination covers two cases: requests that include the header with the wrong value, and requests that omit the header entirely. Using only StringNotEquals would miss the second case.

AWS Config: the detective and corrective layer

For buckets that already exist, Config evaluates each one against the s3-bucket-server-side-encryption-enabled managed rule. When it finds a non-compliant bucket, it triggers an SSM Automation document that applies SSE-KMS automatically.

No manual intervention. No Lambda to maintain.

Terraform

The files below create the SCP, the delivery bucket for Config, the Config rule with remediation for the management account, and deploy the same Config rule to all member accounts using a StackSet.

File structure

├── main.tf
├── variables.tf
├── config_bucket.tf
├── scp.tf
├── config.tf
├── iam.tf
└── stackset.tf

`variables.tf`

variable "organization_root_id" {
  description = "Root ID of the AWS Organization (e.g. r-xxxx)"
  type        = string
}

variable "kms_key_arn" {
  description = "ARN of the KMS key to use for SSE-KMS enforcement"
  type        = string
}

`scp.tf`: create and attach the SCP

resource "aws_organizations_policy" "s3_require_ssekms" {
  name        = "RequireS3SSEKMS"
  description = "Denies creation of S3 buckets without SSE-KMS"
  type        = "SERVICE_CONTROL_POLICY"

  content = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Sid      = "DenyS3CreateBucketWithoutSSEKMS"
        Effect   = "Deny"
        Action   = "s3:CreateBucket"
        Resource = "*"
        Condition = {
          StringNotEqualsIfExists = {
            "s3:x-amz-server-side-encryption" = "aws:kms"
          }
          Null = {
            "s3:x-amz-server-side-encryption" = "true"
          }
        }
      }
    ]
  })
}

resource "aws_organizations_policy_attachment" "s3_ssekms_root" {
  policy_id = aws_organizations_policy.s3_require_ssekms.id
  target_id = var.organization_root_id
}

Attaching to the root OU means every account in the organization is covered, including accounts added in the future.

`iam.tf`: roles for Config and SSM remediation

Two roles are needed. The first allows Config to record and deliver configuration data. The second allows SSM Automation to call s3:PutEncryptionConfiguration during remediation. Without this second role, auto-remediation will trigger but fail silently. Config marks the execution as failed and retries up to the maximum_automatic_attempts limit.

# Role for AWS Config recorder
resource "aws_iam_role" "config_role" {
  name = "AWSConfigRole"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Action    = "sts:AssumeRole"
      Effect    = "Allow"
      Principal = { Service = "config.amazonaws.com" }
    }]
  })
}

resource "aws_iam_role_policy_attachment" "config_managed" {
  role       = aws_iam_role.config_role.name
  policy_arn = "arn:aws:iam::aws:policy/service-role/AWS_ConfigRole"
}

# Role for SSM Automation remediation
resource "aws_iam_role" "ssm_remediation_role" {
  name = "SSMRemediationS3SSEKMSRole"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Action    = "sts:AssumeRole"
      Effect    = "Allow"
      Principal = { Service = "ssm.amazonaws.com" }
    }]
  })
}

resource "aws_iam_role_policy" "ssm_remediation_policy" {
  name = "SSMRemediationS3SSEKMSPolicy"
  role = aws_iam_role.ssm_remediation_role.id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Sid    = "AllowS3Encryption"
        Effect = "Allow"
        Action = [
          "s3:PutEncryptionConfiguration",
          "s3:GetEncryptionConfiguration"
        ]
        Resource = "*"
      },
      {
        Sid    = "AllowKMSDescribe"
        Effect = "Allow"
        Action = [
          "kms:DescribeKey"
        ]
        Resource = var.kms_key_arn
      }
    ]
  })
}

`config.tf` - Config rule and remediation for the management account

If you also want to enforce this on the management account itself, add the rule directly. Member accounts are handled via the StackSet below.

resource "aws_config_config_rule" "s3_sse_enabled" {
  name = "s3-bucket-server-side-encryption-enabled"

  source {
    owner             = "AWS"
    source_identifier = "S3_BUCKET_SERVER_SIDE_ENCRYPTION_ENABLED"
  }

  depends_on = [aws_config_configuration_recorder_status.main]
}

resource "aws_config_remediation_configuration" "s3_sse_remediation" {
  config_rule_name = aws_config_config_rule.s3_sse_enabled.name
  target_type      = "SSM_DOCUMENT"
  target_id        = "AWS-EnableS3BucketEncryption"

  automatic                  = true
  maximum_automatic_attempts = 3
  retry_attempt_seconds      = 60

  # The role ARN is required - without it SSM has no permissions to act
  resource_type = "AWS::S3::Bucket"

  parameter {
    name           = "AutomationAssumeRole"
    static_value   = aws_iam_role.ssm_remediation_role.arn
  }

  parameter {
    name           = "BucketName"
    resource_value = "RESOURCE_ID"
  }

  parameter {
    name         = "SSEAlgorithm"
    static_value = "aws:kms"
  }

  parameter {
    name         = "KMSMasterKeyID"
    static_value = var.kms_key_arn
  }

  execution_controls {
    ssm_controls {
      concurrent_execution_rate_percentage = 25
      error_percentage                     = 20
    }
  }
}

resource "aws_config_configuration_recorder" "main" {
  name     = "default"
  role_arn = aws_iam_role.config_role.arn

  recording_group {
    all_supported = true
  }
}

resource "aws_config_delivery_channel" "main" {
  name           = "default"
  s3_bucket_name = aws_s3_bucket.config_delivery.bucket
  depends_on     = [aws_config_configuration_recorder.main]
}

resource "aws_config_configuration_recorder_status" "main" {
  name       = aws_config_configuration_recorder.main.name
  is_enabled = true
  depends_on = [aws_config_delivery_channel.main]
}

`stackset.tf`: deploy Config rule to all member accounts

CloudFormation StackSets let you deploy the Config rule and remediation to every member account without logging into each one. The AutomationAssumeRole in the template body references a role that must exist in each member account. The StackSet creates it via CAPABILITY_NAMED_IAM.

resource "aws_cloudformation_stack_set" "config_s3_ssekms" {
  name             = "ConfigS3SSEKMSRule"
  permission_model = "SERVICE_MANAGED"
  description      = "Deploys AWS Config rule to enforce SSE-KMS on S3 buckets"

  auto_deployment {
    enabled                          = true
    retain_stacks_on_account_removal = false
  }

  capabilities = ["CAPABILITY_NAMED_IAM"]

  template_body = jsonencode({
    AWSTemplateFormatVersion = "2010-09-09"
    Parameters = {
      KMSKeyArn = {
        Type        = "String"
        Description = "ARN of the KMS key for SSE-KMS"
      }
    }
    Resources = {
      SSMRemediationRole = {
        Type = "AWS::IAM::Role"
        Properties = {
          RoleName = "SSMRemediationS3SSEKMSRole"
          AssumeRolePolicyDocument = {
            Version = "2012-10-17"
            Statement = [{
              Effect    = "Allow"
              Principal = { Service = "ssm.amazonaws.com" }
              Action    = "sts:AssumeRole"
            }]
          }
          Policies = [{
            PolicyName = "SSMRemediationS3SSEKMSPolicy"
            PolicyDocument = {
              Version = "2012-10-17"
              Statement = [
                {
                  Effect   = "Allow"
                  Action   = ["s3:PutEncryptionConfiguration", "s3:GetEncryptionConfiguration"]
                  Resource = "*"
                },
                {
                  Effect   = "Allow"
                  Action   = ["kms:DescribeKey"]
                  Resource = { Ref = "KMSKeyArn" }
                }
              ]
            }
          }]
        }
      }
      S3SSEKMSConfigRule = {
        Type = "AWS::Config::ConfigRule"
        Properties = {
          ConfigRuleName = "s3-bucket-server-side-encryption-enabled"
          Source = {
            Owner            = "AWS"
            SourceIdentifier = "S3_BUCKET_SERVER_SIDE_ENCRYPTION_ENABLED"
          }
        }
      }
      S3SSEKMSRemediation = {
        Type      = "AWS::Config::RemediationConfiguration"
        DependsOn = ["S3SSEKMSConfigRule"]
        Properties = {
          ConfigRuleName           = "s3-bucket-server-side-encryption-enabled"
          TargetType               = "SSM_DOCUMENT"
          TargetId                 = "AWS-EnableS3BucketEncryption"
          Automatic                = true
          MaximumAutomaticAttempts = 3
          RetryAttemptSeconds      = 60
          ResourceType             = "AWS::S3::Bucket"
          Parameters = {
            AutomationAssumeRole = {
              StaticValue = {
                Values = [{
                  "Fn::GetAtt" = ["SSMRemediationRole", "Arn"]
                }]
              }
            }
            BucketName    = { ResourceValue = { Value = "RESOURCE_ID" } }
            SSEAlgorithm  = { StaticValue   = { Values = ["aws:kms"] } }
            KMSMasterKeyID = { StaticValue  = { Values = [{ Ref = "KMSKeyArn" }] } }
          }
        }
      }
    }
  })

  parameters {
    parameter_key   = "KMSKeyArn"
    parameter_value = var.kms_key_arn
  }
}

resource "aws_cloudformation_stack_set_instance" "config_s3_ssekms_org" {
  stack_set_name = aws_cloudformation_stack_set.config_s3_ssekms.name

  deployment_targets {
    organizational_unit_ids = [var.organization_root_id]
  }

  operation_preferences {
    failure_tolerance_percentage = 20
    max_concurrent_percentage    = 25
  }
}

auto_deployment enabled means that any new account added to the organization gets the Config rule and SSM role deployed automatically.

What happens after deployment?

New bucket without SSE-KMS -> s3:CreateBucket hits the SCP condition -> AccessDenied. The bucket is never created.

New bucket with SSE-KMS -> passes the SCP condition -> created normally.

Existing bucket without SSE-KMS -> Config evaluates it as non-compliant -> SSM Automation triggers using SSMRemediationS3SSEKMSRole -> AWS-EnableS3BucketEncryption calls s3:PutEncryptionConfiguration -> bucket becomes compliant. This happens within minutes of the rule being active.

Validating the deployment

Three checks to run after terraform apply completes.

Check 1: SCP is blocking unencrypted bucket creation

From any member account (not the management account, which is excluded from SCPs):

# This should fail with AccessDenied
aws s3api create-bucket \
  --bucket test-no-encryption-$(date +%s) \
  --region us-east-1

# Expected output:
# An error occurred (AccessDenied) when calling the CreateBucket operation

# This should succeed
aws s3api create-bucket \
  --bucket test-with-encryption-$(date +%s) \
  --region us-east-1 \
  --create-bucket-configuration LocationConstraint=us-east-1 \
  --object-lock-enabled-for-bucket

aws s3api put-bucket-encryption \
  --bucket test-with-encryption-<timestamp> \
  --server-side-encryption-configuration '{
    "Rules": [{
      "ApplyServerSideEncryptionByDefault": {
        "SSEAlgorithm": "aws:kms",
        "KMSMasterKeyID": "<your-kms-key-arn>"
      }
    }]
  }'

Check 2: Config rule is active and evaluating

aws configservice describe-config-rules \
  --config-rule-names s3-bucket-server-side-encryption-enabled \
  --query 'ConfigRules[0].ConfigRuleState'
# Expected: "ACTIVE"

Check 3: compliance status for existing buckets

aws configservice get-compliance-details-by-config-rule \
  --config-rule-name s3-bucket-server-side-encryption-enabled \
  --compliance-types NON_COMPLIANT \
  --query 'EvaluationResults[*].EvaluationResultIdentifier.EvaluationResultQualifier.ResourceId'

Non-compliant buckets appearing here mean the SSM Automation has not run yet. Wait a few minutes and run the query again. To trigger remediation immediately:

aws configservice start-remediation-execution \
  --config-rule-name s3-bucket-server-side-encryption-enabled \
  --resource-keys '[{"resourceType":"AWS::S3::Bucket","resourceId":"<bucket-name>"}]'

A note on the KMS key

The KMSMasterKeyID parameter accepts either a key ARN or an alias. In a multi-account setup, you have two options:

Use a single centralized KMS key in the management account and share it via key policy with member accounts
Create one KMS key per account and pass the correct ARN per account in the StackSet parameter overrides

A centralized key with cross-account access is simpler to manage. If data sovereignty or account isolation is a concern, per-account keys give you finer control.

Cost considerations

AWS Config charges per configuration item recorded. Every time a resource changes state, Config records it. With many accounts and many buckets, the total is worth knowing before you deploy.

Current pricing:

Config rule evaluations: $0.001 per evaluation
Configuration items recorded: $0.003 per item in the first 100K per month, per region

Example: 50 accounts × 20 buckets × 10 configuration changes per bucket per month = 10,000 configuration items, around $30/month across the organization. The number grows with bucket churn, so if your environment creates and deletes buckets frequently, pull a Config cost report after the first month.

The SCP itself has no cost.

Why not the other options?

Option 1 (CloudFormation StackSets + Config only, no SCP) covers existing buckets but does nothing to prevent new ones from being created without encryption. You are always reacting.

Option 2 (Security Hub + EventBridge + Lambda) works, but you are now maintaining Lambda code, IAM roles for that Lambda, and EventBridge rules per account. More moving parts, more things that can break silently.

Option 3 (bucket policies via StackSets + Drift Detection) requires that the bucket already exists before you can apply a policy to it. It does not prevent the creation of unencrypted buckets. It adds a policy after the fact. Drift Detection is manual.

The SCP + Config combination uses two managed AWS mechanisms, no custom code, and applies automatically to new accounts.

See you next time, right? Bye.

How I would put an SLA operations portal on AWS (and what I weighed along the way)

Carlos Filho — Tue, 02 Jun 2026 20:30:00 +0000

Context

I maintain an internal portal that tracks SLA across several operational queues in Jira. It reads tickets through the Jira API, hours through the Tempo API, calculates a health score per team, and shows everything in a dashboard. Today it runs as a simple Flask server, with an in memory cache and a few JSON files to keep history.

It works. But "runs on my machine" is not architecture. So I sat down to think: if I needed to ship this for real, with people depending on it, what would it look like on AWS?

This post is that exercise. It is not a step by step Terraform tutorial. It is more about the decisions and the trade-offs. I will try to be honest about what makes sense and what would be overkill.

Portal home screen: consolidated health score and per team.

What the portal does today

Before throwing AWS on top of it, it helps to understand the parts:

Backend: a Flask server in Python. It exposes a few REST endpoints (/api/queues/:id/data, /api/queues/overview, /api/trends, and so on).
Frontend: a single HTML file with a fair amount of JavaScript and Chart.js. No framework, no build step.
External integrations: the Jira REST API and the Tempo API. That is where all the data comes from.
Cache: an in memory dictionary with a 5 minute TTL. The first call to a large queue (like Service Desk, with around 2,500 tickets) takes about 30 seconds; the next ones come back instantly.
Persistence: two JSON files. One is the queue registry (configuration), the other keeps daily snapshots of the health score to build trend charts.
Daily job: once a day it takes a "snapshot" of all queues and saves it to history.
Secrets: Jira and Tempo tokens, today in environment variables.

Notice that it is a small system. That matters, because the temptation to throw ten AWS services at a project this size is strong and almost always wrong.

Operations page for one queue: first response and resolution SLA, multidimensional health score, and distribution by analyst.

The AWS design

I will split it by responsibility.

Where the backend runs

The first question is: Lambda or container?

The backend has one annoying detail. The call that fetches all tickets from a large queue can take more than 30 seconds because of Jira API pagination. Lambda has a maximum timeout of 15 minutes, so it would fit, but 30 seconds of cold start plus a synchronous call that size makes me uncomfortable for a UI. Nobody likes clicking and waiting half a minute.

So I went with ECS Fargate. A container running Flask (behind Gunicorn, not the Flask dev server), running as a service. I do not have to manage EC2, I pay for what I use, and I do not have the cold start dance for the heavier requests.

I put an Application Load Balancer in front. It handles HTTPS (with an ACM certificate) and distributes across the containers. If usage grows one day, I turn on Fargate auto scaling by CPU. Today, with the current team, one or two tasks are enough.

One alternative I considered was App Runner. It is simpler than plain Fargate and it abstracts away the load balancer. For a project this size, App Runner would probably be enough and would give me less to configure. I stayed with Fargate because I know it better and I want control over the networking, but if I were in a hurry, App Runner would be a defensible choice.

The frontend

The frontend is static HTML. It does not need a server for that.

I drop the file in an S3 bucket and put CloudFront in front. CloudFront caches the HTML close to the user and serves it over HTTPS. Flask stops having the job of serving the HTML and becomes only an API.

A practical detail: since the frontend calls the API at /api/..., I configure CloudFront with two origins. A request that hits /api/* goes to the Load Balancer (the backend); everything else comes from S3. That way the user only knows one domain and I do not have to worry about CORS.

Persistence

This is where I had to hold back the most so I would not overcomplicate things.

The two JSON files have different natures:

Queue registry (queue_registry.json): it is configuration. It rarely changes, and by hand. This can stay a file, just in an S3 bucket instead of local disk. The backend reads it at startup (or with a short cache). I do not need a database for this.

Snapshot history (trends.json): this grows every day and I run queries like "give me the last 30 days of this queue." A single JSON file that keeps growing is a bad idea in the medium term, because every time I read the whole file just to grab a slice. Here DynamoDB fits well. Each snapshot becomes an item, with the date as part of the key. I can fetch a date range without reading the entire history. And it is serverless, so I do not have a database instance to babysit on a small project.

I noticed I could have gone with RDS, but I have no complex relationships and no JOINs. It would mean paying for something I do not use. DynamoDB with a simple access pattern (queue key plus date) does the job.

The cache

Today the cache lives in process memory. It works because it is a single process. The moment I have two tasks on Fargate, each one has its own cache, and that is where the inconsistency starts. One user gets data from 1 minute ago, another from 4 minutes ago.

The standard answer for this is ElastiCache (Redis). A cache shared across the tasks, with a TTL. The tasks write to and read from the same place.

But, and here is where I would be honest in a real article, for this volume I might not even stand up Redis on day one. Keeping the cache in memory with two tasks means that, in the worst case, the data can be 5 minutes "out of phase" between one task and another. For an SLA dashboard that refreshes every 5 minutes anyway, that is tolerable. I would add Redis when the number of tasks grows or when the cost of re fetching from Jira starts to hurt. It is one of those things you can put off without shame.

The daily job

The daily snapshot is a cron today. On AWS that becomes EventBridge Scheduler firing once a day.

What does it fire? Here Lambda makes sense. The job does not have the UI latency problem. It runs in the background, it can take the 30 to 40 seconds fetching all the queues, and nobody is waiting on a screen. A Lambda function that takes the snapshot and writes to DynamoDB. Cheap, with no server running all day just to run once.

Secrets

The Jira and Tempo tokens cannot sit in a plain text environment variable in a serious setup. They go to Secrets Manager. The container and the Lambda pull the tokens at runtime through an IAM role. Bonus: Secrets Manager does rotation if we configure it, although a third party API token does not always support automatic rotation.

Logs and the minimum of observability

Everything sends logs to CloudWatch Logs. That comes almost for free with Fargate and Lambda.

I would set at least two alarms in CloudWatch:

If the 5xx error rate on the Load Balancer rises, I want to know.
If the daily job fails (the Lambda errors out), I want to know, because otherwise the trend chart gets a gap and I only find out weeks later.

I am not going to build an elaborate observability dashboard for a project this size. The CloudWatch logs and two alarms cover what matters.

Putting it all together

In the AWS version, the user would reach the portal through a domain like this:

https://d1a2b3c4d5e6f7.cloudfront.net

(that is the domain CloudFront generates automatically; in production I would point a CNAME like cldops.mycompany.com to it through Route 53)

The flow looks like this:

User reaches https://d1a2b3c4d5e6f7.cloudfront.net, hits CloudFront.
CloudFront serves the HTML from S3.
The JavaScript calls /api/..., CloudFront routes it to the Load Balancer, then to Fargate (Flask/Gunicorn).
Flask checks the cache (memory or Redis), and if needed, fetches from Jira/Tempo.
Queue configuration comes from a JSON in S3; history comes from DynamoDB.
Once a day, EventBridge fires a Lambda that takes the snapshot and writes to DynamoDB.
Tokens live in Secrets Manager; logs and alarms in CloudWatch.

As a box diagram, it looks roughly like this:

                          ┌─────────────────┐
   user ────HTTPS───────► │   CloudFront     │
                          │ d1a2b3c4d5e6f7   │
                          │ .cloudfront.net  │
                          └────────┬─────────┘
                          /            \
                   /api/* │             │ rest
                          ▼             ▼
                  ┌──────────────┐   ┌──────────┐
                  │     ALB      │   │    S3     │
                  └──────┬───────┘   │ (HTML)    │
                         ▼           └──────────┘
                  ┌──────────────┐
                  │ ECS Fargate  │──► Jira API
                  │ Flask/Gunic. │──► Tempo API
                  └──────┬───────┘
                         │
              ┌──────────┼──────────┐
              ▼          ▼          ▼
        ┌─────────┐ ┌─────────┐ ┌──────────┐
        │DynamoDB │ │  S3     │ │ Secrets  │
        │(trends) │ │(config) │ │ Manager  │
        └─────────┘ └─────────┘ └──────────┘

   EventBridge (1x/day) ──► Lambda (snapshot) ──► DynamoDB

The cross queue comparison: you can see at a glance which queue is healthy and which one needs attention.

How this becomes IaC

A diagram is nice, but it does not deploy anything. I would write this in Terraform so I do not end up clicking around the console and forgetting what I did. I am not going to paste the whole module here (it would be huge), but I can show the skeleton so you get a sense of the size.

The file structure I would use:

infra/
├── main.tf            # providers and state backend
├── variables.tf       # region, project name, etc
├── network.tf         # VPC, subnets, security groups
├── ecs.tf             # cluster, task definition, service
├── alb.tf             # load balancer and target group
├── frontend.tf        # S3 bucket + CloudFront
├── data.tf            # DynamoDB + config bucket
├── scheduler.tf       # EventBridge + snapshot Lambda
├── secrets.tf         # Secrets Manager
└── outputs.tf         # CloudFront URL, etc

A piece of frontend.tf, which is what generates that CloudFront domain:

# Bucket that holds the dashboard HTML
resource "aws_s3_bucket" "frontend" {
  bucket = "${var.project}-frontend"
}

resource "aws_cloudfront_distribution" "portal" {
  enabled             = true
  default_root_object = "dashboard.html"

  # Origin 1: the static HTML in S3
  origin {
    domain_name              = aws_s3_bucket.frontend.bucket_regional_domain_name
    origin_id                = "s3-frontend"
    origin_access_control_id = aws_cloudfront_origin_access_control.frontend.id
  }

  # Origin 2: the API on the load balancer
  origin {
    domain_name = aws_lb.api.dns_name
    origin_id   = "alb-api"
    custom_origin_config {
      http_port              = 80
      https_port             = 443
      origin_protocol_policy = "https-only"
      origin_ssl_protocols   = ["TLSv1.2"]
    }
  }

  # Everything that is NOT /api/* comes from S3
  default_cache_behavior {
    target_origin_id       = "s3-frontend"
    viewer_protocol_policy = "redirect-to-https"
    allowed_methods        = ["GET", "HEAD"]
    cached_methods         = ["GET", "HEAD"]
    cache_policy_id        = data.aws_cloudfront_cache_policy.optimized.id
  }

  # /api/* goes to the backend, no cache (dynamic data)
  ordered_cache_behavior {
    path_pattern           = "/api/*"
    target_origin_id       = "alb-api"
    viewer_protocol_policy = "redirect-to-https"
    allowed_methods        = ["GET", "HEAD", "OPTIONS", "POST"]
    cached_methods         = ["GET", "HEAD"]
    cache_policy_id        = data.aws_cloudfront_cache_policy.disabled.id
  }

  restrictions {
    geo_restriction { restriction_type = "none" }
  }

  viewer_certificate {
    cloudfront_default_certificate = true
  }
}

output "portal_url" {
  value = "https://${aws_cloudfront_distribution.portal.domain_name}"
}

Notice the detail with the two cache_behavior blocks: I let CloudFront cache the static HTML freely, but I mark /api/* as no cache, because it is dynamic SLA data. If I cached the API at CloudFront, it would fight with the 5 minute cache that already exists in the backend and the data would go stale in ways that are hard to debug.

And outputs.tf prints the URL at the end of terraform apply:

# after the apply:
# portal_url = "https://d1a2b3c4d5e6f7.cloudfront.net"

A snippet of ecs.tf to give an idea of how the backend is declared:

resource "aws_ecs_task_definition" "api" {
  family                   = "${var.project}-api"
  requires_compatibilities = ["FARGATE"]
  network_mode             = "awsvpc"
  cpu                      = 512
  memory                   = 1024
  execution_role_arn       = aws_iam_role.ecs_execution.arn
  task_role_arn            = aws_iam_role.ecs_task.arn

  container_definitions = jsonencode([{
    name  = "flask-api"
    image = "${aws_ecr_repository.api.repository_url}:latest"
    portMappings = [{ containerPort = 8080 }]
    # tokens come from Secrets Manager, not hardcoded
    secrets = [
      { name = "JIRA_API_TOKEN", valueFrom = aws_secretsmanager_secret.jira.arn },
      { name = "TEMPO_API_TOKEN", valueFrom = aws_secretsmanager_secret.tempo.arn }
    ]
    logConfiguration = {
      logDriver = "awslogs"
      options = {
        "awslogs-group"         = aws_cloudwatch_log_group.api.name
        "awslogs-region"        = var.region
        "awslogs-stream-prefix" = "flask"
      }
    }
  }])
}

This is not the full file. It is missing the aws_ecs_service, the auto scaling, the IAM policies with the right permission for the secrets, the network.tf with the VPC. But you can see it is not magic: it is declaring each box from the diagram and wiring one to the other by their IDs.

If I were doing this for real, I probably would not even write all of it by hand. I would use a ready made community module for the VPC (the terraform-aws-modules/vpc/aws is well tested) and spend my time on what is specific to the project.

What I left out on purpose

I think half the value of this kind of exercise is in what you do NOT include.

No Kubernetes (EKS). For a single container, EKS would be bringing a tractor to go to the bakery.
No separate API Gateway. The Load Balancer already handles the routing. API Gateway would make sense if I had several APIs, complex authorization, throttling per client. That is not the case.
No data lake, no Athena, no Glue. It is SLA history, not big data. DynamoDB handles it.
No multi region. It is an internal tool. If the region goes down for a few hours, the team looks at Jira directly. It is not worth the complexity.

Rough cost

I am not going to pretend I know the exact number, because it depends a lot on usage. But the order of magnitude for an internal system this size:

Fargate: the biggest item, but with one or two small tasks it is manageable.
S3 + CloudFront: cents for this amount of traffic.
DynamoDB: on demand, paying per write/read, with one snapshot a day and occasional queries, it stays very low.
Lambda + EventBridge: runs once a day, basically free.
Secrets Manager: charges per secret per month, a few dollars.

The bulk of the cost is Fargate running all the time. If usage were very sporadic, then it would be worth reconsidering Lambda for the backend and accepting the cold start.

Migration, if I were to do it for real

I would not do everything at once. The order I would follow:

Containerize Flask and ship it to Fargate behind the Load Balancer. The system keeps using local JSON files inside the container (ephemeral, but it works to start).
Move the frontend to S3 + CloudFront.
Take the tokens out of the environment and put them in Secrets Manager.
Migrate the snapshot history from JSON to DynamoDB. This is the step that actually changes code.
Move the daily job to EventBridge + Lambda.
Only then, if needed, stand up Redis on ElastiCache.

Each step leaves the system working. No big bang.

The trends page uses the daily snapshots, which is exactly the data that would move to DynamoDB.

Wrapping up

The point I wanted to get across is that good architecture is not the one that uses the most services. It is the one that uses enough for the problem you have. This portal is small and will probably stay small. Its AWS version reflects that: Fargate for the backend, S3/CloudFront for the front, DynamoDB for the history, Lambda for the job, and the rest is plumbing (Secrets Manager, CloudWatch, IAM).

If it ever grows to dozens of teams and thousands of hits a day, I will revisit it. But designing for that hypothetical future now would mean spending time and money on a problem I do not have yet.

If you have a similar little project, an internal tool that "runs on your machine," I recommend doing this exercise. You learn more by designing the production version of something small than by copying an architecture diagram from a big company.

Comments and corrections are welcome. If you would have done something differently, tell me, especially on the cache part, which is where I was most unsure.

O AWS App Runner deixará de aceitar novos registros em 30 de abril. O Amazon RDS Custom para Oracle será descontinuado, entre outros serviços da AWS.

Carlos Filho — Tue, 28 Apr 2026 15:50:34 +0000

Nos últimos dias, a AWS anunciou uma série de mudanças no ciclo de vida de alguns serviços. À primeira vista, pode parecer só mais um daqueles comunicados técnicos que passam batido no meio de tanta novidade.

Mas quando você olha com um pouco mais de atenção, dá pra perceber que não é somente descontinuação.

Alguns serviços estão entrando em manutenção o que significa que deixam de aceitar novos clientes enquanto outros já têm data definida para deixar de existir.

E dois casos ajudam a entender bem o que está acontecendo.

App Runner: continua funcionando, mas ficou pelo caminho

O AWS App Runner entra em modo de manutenção a partir de 30 de abril de 2026. Isso quer dizer que novos clientes não conseguem mais utilizar o serviço, embora quem já esteja usando continue normalmente. Esse tipo de movimento é curioso pois o serviço não foi encerrado, ele simplesmente deixará de evoluir.

Fica ali, funcionando, mas sem novas funcionalidades, sem avanço. Com o tempo, naturalmente, começa a perder espaço.

RDS Custom for Oracle: aqui o caminho já está definido

No caso do RDS Custom for Oracle, a situação é mais direta.
A AWS já anunciou que o serviço será encerrado em 31 de março de 2027. Depois disso, não será mais possível acessar instâncias, snapshots ou qualquer recurso relacionado.

Até lá, o serviço continua operando normalmente. Mas a própria AWS já orienta que os clientes planejem a migração, com recomendação clara de saída para ambientes em EC2.

O papel que o RDS Custom ocupava na AWS

Esse serviço sempre existiu para resolver um problema específico.

Ele permitia usar um modelo gerenciado, como o RDS, mas sem abrir mão do acesso ao sistema operacional e ao próprio banco.
Isso fazia sentido para aplicações legadas ou cenários mais sensíveis, onde não dava para trabalhar dentro das limitações do modelo padrão.

Ao mesmo tempo, criava uma situação um pouco desconfortável:

Não era totalmente gerenciado
Nem totalmente controlado pelo cliente

E isso costuma trazer mais complexidade do que parece no início.

O que esses dois casos têm em comum?

Quando você coloca App Runner e RDS Custom lado a lado, começa a aparecer um padrão. São serviços que ocupam um espaço intermediário.

De um lado, o App Runner simplifica bastante a operação. Do outro, o RDS Custom dá liberdade além do padrão gerenciado.
Mas nenhum dos dois se encaixa exatamente no modelo mais comum que a AWS vem reforçando. E, aos poucos, esse tipo de solução vai sendo deixado de lado.

Um movimento que aparece com frequência

Esse tipo de ajuste não é novo.

Plataformas grandes, com muitos serviços, precisam revisar o portfólio de tempos em tempos. Nem tudo continua fazendo sentido conforme o uso real evolui.

O que chama a atenção aqui é a direção. Cada vez menos espaço para soluções “no meio do caminho”.

Ou você segue um modelo mais gerenciado, com menos preocupação operacional. Ou assume o controle completo do ambiente.

E o impacto disso no dia a dia

Esse tipo de mudança costuma se manifestar de forma mais concreta em outro momento.

Quando um ambiente já está rodando há algum tempo. Uma escolha feita lá atrás começa a ser revisitada para saber se é a melhor forma de operar. E aí surgem coisas que não estavam no radar inicial:

necessidade de migração
ajustes de arquitetura
revisão de operação

O que vemos aqui não é somente descontinuação

Esses anúncios acabam sendo mais úteis quando vistos como sinal pois nos mostram o que continua sendo incentivado dentro da plataforma e o que está ficando para trás.

E isso acaba influenciando bastante as decisões que estão sendo tomadas agora, principalmente quando alguém está desenhando algo novo.

Porque, mais do que saber o que existe hoje, o que realmente pesa é entender o que ainda vai fazer sentido amanhã.

Referência: https://aws.amazon.com/about-aws/whats-new/2026/03/aws-service-availability/

That's all folks.

A AWS lançou um ambiente de testes onde você pode experimentar o Aurora DSQL imediatamente, sem precisar fazer login.

Carlos Filho — Mon, 09 Mar 2026 13:42:52 +0000

A Amazon Web Services (AWS) anunciou o lançamento de um ambiente de testes onde você pode experimentar seu serviço de banco de dados, o Amazon Aurora DSQL .

O Amazon Aurora DSQL é um banco de dados distribuído de grande escala com capacidade de escalabilidade praticamente ilimitada e compatibilidade com PostgreSQL. Foi lançado oficialmente em maio de 2025.

Referência: A AWS lança oficialmente o "Amazon Aurora DSQL", um banco de dados distribuído em larga escala compatível com o PostgreSQL e que escala praticamente sem limites.

É descrito como um banco de dados compatível com PostgreSQL que não apenas atinge alta escalabilidade por meio de processamento distribuído e alta disponibilidade por meio de configurações redundantes usando múltiplas zonas e regiões de disponibilidade, mas também supera a alta latência que era considerada uma fraqueza ao se obter forte consistência por meio do processamento de transações em um banco de dados amplamente distribuído, como em múltiplas regiões, e combina alta velocidade com baixa latência.

Verifique facilmente as incompatibilidades entre o Amazon DSQL e o PostgreSQL.

No entanto, devido às diferenças de arquitetura resultantes de sua implementação como um banco de dados distribuído em larga escala, o Aurora DSQL não é totalmente compatível com o PostgreSQL, e existem incompatibilidades em áreas como definição de esquema de banco de dados, criação de índices e processamento de transações .

Este ambiente de testes foi criado para permitir que os desenvolvedores verifiquem facilmente essas incompatibilidades.

Não é necessário cadastro, você pode experimentar a amostra imediatamente.

(Imagem do https://aws.amazon.com/rds/aurora/dsql/)

A tela inicial é mostrada abaixo. Você pode optar por experimentar um banco de dados vazio ou um dos dois bancos de dados de exemplo fornecidos.

Se você selecionar a terceira opção, "Chinook", verá que as configurações de esquema e exemplos de instruções SQL para inserção de dados são fornecidos.

Após executar este comando, selecione "REPL" na tela de consulta e execute uma instrução SQL (por exemplo, select * from artists;), os resultados serão exibidos imediatamente.

That's all folks.

Solving the problem of detecting deviations in multi-account environments using AWS Config and EventBridge

Carlos Filho — Tue, 11 Nov 2025 17:56:58 +0000

Hello guys! I am back after some months working on this project. So, let's go!

The real problem

Working with a client that manages several different AWS accounts, I encountered a critical situation: resources being manually modified outside of CloudFormation in production. In practice, this meant that what was in the code no longer reflected the reality of the infrastructure.

What is not always evident is that this “configuration drift” can cause:

Rollback failures
Inconsistencies between environments
Compliance issues
Difficulty tracking changes

The Solution: Centralized drift detection system

I created an architecture that monitors drift in real-time across multiple AWS accounts, utilizing AWS Config, EventBridge, and SNS for immediate notification.

Let's go to the implementation

Account Hub (Centralized Account)

# hub-account-template.yaml
AWSTemplateFormatVersion: '2010-09-09'
Description: 'Drift Detection Hub - Central Monitoring Account'

Parameters:
  SpokeAccountIds:
    Type: CommaDelimitedList
    Description: List of spoke account IDs to monitor
    Default: "123456789012,987654321098"

  NotificationEmail:
    Type: String
    Description: Email for drift notifications

Resources:
  # Central Event Bus to receive events from all accounts
  CentralEventBus:
    Type: AWS::Events::EventBus
    Properties:
      Name: DriftDetectionCentralBus

  # Permission for spoke accounts to send events
  EventBusPermission:
    Type: AWS::Events::Permission
    Properties:
      StatementId: AllowSpokeAccounts
      EventBusName: !Ref CentralEventBus
      Principal: "*"
      Action: events:PutEvents
      Condition:
        StringEquals:
          "aws:PrincipalOrgID": !Sub "${AWS::AccountId}"

  # DynamoDB for storing drift history
  DriftHistoryTable:
    Type: AWS::DynamoDB::Table
    Properties:
      TableName: drift-detection-history
      BillingMode: PAY_PER_REQUEST
      AttributeDefinitions:
        - AttributeName: stackId
          AttributeType: S
        - AttributeName: timestamp
          AttributeType: N
      KeySchema:
        - AttributeName: stackId
          KeyType: HASH
        - AttributeName: timestamp
          KeyType: RANGE
      StreamSpecification:
        StreamViewType: NEW_AND_OLD_IMAGES
      TimeToLiveSpecification:
        AttributeName: ttl
        Enabled: true

  # Lambda for processing drift events
  DriftProcessorFunction:
    Type: AWS::Lambda::Function
    Properties:
      FunctionName: drift-processor
      Runtime: python3.11
      Handler: index.handler
      Timeout: 60
      Environment:
        Variables:
          TABLE_NAME: !Ref DriftHistoryTable
          SNS_TOPIC_ARN: !Ref AlertTopic
      Role: !GetAtt DriftProcessorRole.Arn
      Code:
        ZipFile: |
          import json
          import boto3
          import os
          from datetime import datetime
          from decimal import Decimal

          dynamodb = boto3.resource('dynamodb')
          sns = boto3.client('sns')
          table = dynamodb.Table(os.environ['TABLE_NAME'])

          def handler(event, context):
              print(f"Received event: {json.dumps(event)}")

              detail = event['detail']
              account_id = event['account']
              region = event['region']

              # Processar drift detection
              stack_id = detail['stack-id']
              stack_name = detail['stack-name']
              drift_status = detail['drift-status']

              # Timestamp for sorting
              timestamp = Decimal(str(datetime.utcnow().timestamp()))

              # Store in DynamoDB
              item = {
                  'stackId': stack_id,
                  'timestamp': timestamp,
                  'accountId': account_id,
                  'region': region,
                  'stackName': stack_name,
                  'driftStatus': drift_status,
                  'driftedResources': detail.get('drifted-resources', []),
                  'ttl': int(timestamp + 2592000)  # 30 days TTL
              }

              table.put_item(Item=item)

              # If there is drift, send alert
              if drift_status == 'DRIFTED':
                  message = f"""
                  ⚠️ DRIFT DETECTED

                  Account: {account_id}
                  Region: {region}
                  Stack: {stack_name}

                  Drifted Resources:
                  {json.dumps(detail.get('drifted-resources', []), indent=2)}

                  Action Required: Review and update CloudFormation template or revert manual changes.
                  """

                  sns.publish(
                      TopicArn=os.environ['SNS_TOPIC_ARN'],
                      Subject=f'🚨 Drift Detected: {stack_name}',
                      Message=message
                  )

              return {'statusCode': 200}

  DriftProcessorRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service: lambda.amazonaws.com
            Action: sts:AssumeRole
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
      Policies:
        - PolicyName: DriftProcessorPolicy
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action:
                  - dynamodb:PutItem
                  - dynamodb:GetItem
                  - dynamodb:Query
                Resource: !GetAtt DriftHistoryTable.Arn
              - Effect: Allow
                Action:
                  - sns:Publish
                Resource: !Ref AlertTopic

  # EventBridge Rule to capture drift events
  DriftDetectionRule:
    Type: AWS::Events::Rule
    Properties:
      Name: capture-drift-events
      EventBusName: !Ref CentralEventBus
      EventPattern:
        source:
          - custom.drift-detection
        detail-type:
          - Stack Drift Detected
      State: ENABLED
      Targets:
        - Arn: !GetAtt DriftProcessorFunction.Arn
          Id: drift-processor

  # Permission for EventBridge to invoke Lambda
  LambdaInvokePermission:
    Type: AWS::Lambda::Permission
    Properties:
      FunctionName: !Ref DriftProcessorFunction
      Action: lambda:InvokeFunction
      Principal: events.amazonaws.com
      SourceArn: !GetAtt DriftDetectionRule.Arn

  # SNS Topic for alerts
  AlertTopic:
    Type: AWS::SNS::Topic
    Properties:
      TopicName: drift-detection-alerts
      Subscription:
        - Endpoint: !Ref NotificationEmail
          Protocol: email

Outputs:
  CentralEventBusArn:
    Value: !GetAtt CentralEventBus.Arn
    Export:
      Name: DriftDetection-CentralBusArn

Spoke Accounts (Monitored Accounts)

# spoke-account-template.yaml
AWSTemplateFormatVersion: '2010-09-09'
Description: 'Drift Detection Spoke - Monitored Account Setup'

Parameters:
  HubAccountId:
    Type: String
    Description: Hub account ID for central monitoring

  MonitoringFrequency:
    Type: String
    Default: rate(6 hours)
    Description: How often to check for drift

Resources:
  # AWS Config to detect changes
  ConfigRecorder:
    Type: AWS::Config::ConfigurationRecorder
    Properties:
      Name: drift-detection-recorder
      RoleArn: !GetAtt ConfigRole.Arn
      RecordingGroup:
        AllSupported: false
        ResourceTypes:
          - AWS::CloudFormation::Stack
          - AWS::EC2::Instance
          - AWS::RDS::DBInstance
          - AWS::Lambda::Function
          - AWS::S3::Bucket

  ConfigDeliveryChannel:
    Type: AWS::Config::DeliveryChannel
    Properties:
      Name: drift-detection-channel
      S3BucketName: !Ref ConfigBucket

  ConfigBucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: !Sub 'config-drift-${AWS::AccountId}-${AWS::Region}'
      BucketEncryption:
        ServerSideEncryptionConfiguration:
          - ServerSideEncryptionByDefault:
              SSEAlgorithm: AES256
      VersioningConfiguration:
        Status: Enabled
      LifecycleConfiguration:
        Rules:
          - Id: DeleteOldFiles
            Status: Enabled
            ExpirationInDays: 90

  ConfigRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service: config.amazonaws.com
            Action: sts:AssumeRole
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/ConfigRole
      Policies:
        - PolicyName: S3Access
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action:
                  - s3:PutObject
                  - s3:GetBucketAcl
                Resource:
                  - !Sub '${ConfigBucket.Arn}/*'
                  - !GetAtt ConfigBucket.Arn

  # Lambda to check drift periodically
  DriftCheckerFunction:
    Type: AWS::Lambda::Function
    Properties:
      FunctionName: drift-checker
      Runtime: python3.11
      Handler: index.handler
      Timeout: 300
      Environment:
        Variables:
          HUB_ACCOUNT_ID: !Ref HubAccountId
          HUB_EVENT_BUS: !Sub 'arn:aws:events:${AWS::Region}:${HubAccountId}:event-bus/DriftDetectionCentralBus'
      Role: !GetAtt DriftCheckerRole.Arn
      Code:
        ZipFile: |
          import boto3
          import json
          import os
          from datetime import datetime

          cfn = boto3.client('cloudformation')
          events = boto3.client('events')

          def handler(event, context):
              print("Starting drift detection scan...")

              # Listar todas as stacks
              paginator = cfn.get_paginator('list_stacks')
              page_iterator = paginator.paginate(
                  StackStatusFilter=['CREATE_COMPLETE', 'UPDATE_COMPLETE']
              )

              detected_drifts = []

              for page in page_iterator:
                  for stack in page['StackSummaries']:
                      stack_name = stack['StackName']
                      print(f"Checking stack: {stack_name}")

                      try:
                          # Start drift detection
                          detection_id = cfn.detect_stack_drift(
                              StackName=stack_name
                          )['StackDriftDetectionId']

                          # Awaiting results
                          waiter = cfn.get_waiter('stack_drift_detection_complete')
                          waiter.wait(
                              StackName=stack_name,
                              StackDriftDetectionId=detection_id,
                              WaiterConfig={'Delay': 5, 'MaxAttempts': 60}
                          )

                          # Get drift status
                          drift_status = cfn.describe_stack_drift_detection_status(
                              StackDriftDetectionId=detection_id
                          )

                          if drift_status['StackDriftStatus'] == 'DRIFTED':
                              print(f"DRIFT DETECTED in {stack_name}")

                              # Obtain resources with drift
                              drifted_resources = []
                              resource_drifts = cfn.describe_stack_resource_drifts(
                                  StackName=stack_name,
                                  StackResourceDriftStatusFilters=['MODIFIED', 'DELETED']
                              )

                              for resource in resource_drifts.get('StackResourceDrifts', []):
                                  drifted_resources.append({
                                      'LogicalId': resource['LogicalResourceId'],
                                      'PhysicalId': resource.get('PhysicalResourceId'),
                                      'ResourceType': resource['ResourceType'],
                                      'DriftStatus': resource['StackResourceDriftStatus'],
                                      'Differences': resource.get('PropertyDifferences', [])
                                  })

                              # Send event to hub account
                              event_detail = {
                                  'stack-id': stack['StackId'],
                                  'stack-name': stack_name,
                                  'drift-status': 'DRIFTED',
                                  'detection-time': datetime.utcnow().isoformat(),
                                  'drifted-resources': drifted_resources
                              }

                              events.put_events(
                                  Entries=[{
                                      'Source': 'custom.drift-detection',
                                      'DetailType': 'Stack Drift Detected',
                                      'Detail': json.dumps(event_detail),
                                      'EventBusName': os.environ['HUB_EVENT_BUS']
                                  }]
                              )

                              detected_drifts.append(stack_name)

                      except Exception as e:
                          print(f"Error checking {stack_name}: {str(e)}")
                          continue

              return {
                  'statusCode': 200,
                  'body': json.dumps({
                      'message': 'Drift detection completed',
                      'drifted_stacks': detected_drifts
                  })
              }

  DriftCheckerRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service: lambda.amazonaws.com
            Action: sts:AssumeRole
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
      Policies:
        - PolicyName: DriftCheckerPolicy
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action:
                  - cloudformation:DetectStackDrift
                  - cloudformation:DescribeStackDriftDetectionStatus
                  - cloudformation:DescribeStackResourceDrifts
                  - cloudformation:ListStacks
                  - cloudformation:DescribeStacks
                Resource: '*'
              - Effect: Allow
                Action:
                  - events:PutEvents
                Resource: !Sub 'arn:aws:events:${AWS::Region}:${HubAccountId}:event-bus/DriftDetectionCentralBus'

  # EventBridge Rule for periodic trigger
  DriftCheckSchedule:
    Type: AWS::Events::Rule
    Properties:
      Name: drift-check-schedule
      ScheduleExpression: !Ref MonitoringFrequency
      State: ENABLED
      Targets:
        - Arn: !GetAtt DriftCheckerFunction.Arn
          Id: drift-checker

  SchedulePermission:
    Type: AWS::Lambda::Permission
    Properties:
      FunctionName: !Ref DriftCheckerFunction
      Action: lambda:InvokeFunction
      Principal: events.amazonaws.com
      SourceArn: !GetAtt DriftCheckSchedule.Arn

Deployment via CLI

# 1. Deploy in Hub Account
aws cloudformation create-stack \
  --stack-name drift-detection-hub \
  --template-body file://hub-account-template.yaml \
  --parameters \
    ParameterKey=NotificationEmail,ParameterValue=sre-team@company.com \
    ParameterKey=SpokeAccountIds,ParameterValue="111111111111,222222222222" \
  --capabilities CAPABILITY_IAM \
  --region us-east-1

# Wait for creation to complete
aws cloudformation wait stack-create-complete \
  --stack-name drift-detection-hub \
  --region us-east-1

# 2. Deploy to each Spoke Account
for ACCOUNT in 111111111111 222222222222; do
  echo "Deploying to account ${ACCOUNT}..."
  aws cloudformation create-stack \
    --stack-name drift-detection-spoke \
    --template-body file://spoke-account-template.yaml \
    --parameters \
      ParameterKey=HubAccountId,ParameterValue=999999999999 \
      ParameterKey=MonitoringFrequency,ParameterValue="rate(4 hours)" \
    --capabilities CAPABILITY_IAM \
    --region us-east-1 \
    --profile account-${ACCOUNT}
done

Actual implementation results

First execution - Initial Discovery

# Manual trigger for immediate testing
aws lambda invoke \
  --function-name drift-checker \
  --invocation-type RequestResponse \
  --payload '{}' \
  --cli-binary-format raw-in-base64-out \
  response.json \
  --profile account-111111111111

cat response.json | jq '.'

Lambda's response:

{
  "statusCode": 200,
  "body": {
    "message": "Drift detection completed",
    "summary": {
      "total_stacks": 23,
      "drifted_stacks": 7,
      "in_sync_stacks": 15,
      "failed_checks": 1
    },
    "drifted_stacks": [
      {
        "stack": "production-api-stack",
        "resources": 3
      },
      {
        "stack": "database-cluster-prod",
        "resources": 1
      },
      {
        "stack": "frontend-cdn-stack",
        "resources": 2
      },
      {
        "stack": "monitoring-stack",
        "resources": 4
      },
      {
        "stack": "auth-service-stack",
        "resources": 2
      },
      {
        "stack": "data-pipeline-stack",
        "resources": 1
      },
      {
        "stack": "backup-automation-stack",
        "resources": 2
      }
    ]
  }
}

Specific Drift Details

aws cloudformation describe-stack-resource-drifts \
  --stack-name production-api-stack \
  --stack-resource-drift-status-filters MODIFIED DELETED \
  --profile account-111111111111

Detailed response

{
  "StackResourceDrifts": [
    {
      "StackId": "arn:aws:cloudformation:us-east-1:111111111111:stack/production-api-stack/abc123",
      "LogicalResourceId": "APIGatewayRestApi",
      "PhysicalResourceId": "api-gw-prod-123",
      "ResourceType": "AWS::ApiGateway::RestApi",
      "ExpectedProperties": {
        "EndpointConfiguration": {
          "Types": ["REGIONAL"]
        },
        "MinimumCompressionSize": 1024
      },
      "ActualProperties": {
        "EndpointConfiguration": {
          "Types": ["EDGE"]
        },
        "MinimumCompressionSize": 2048
      },
      "PropertyDifferences": [
        {
          "PropertyPath": "/EndpointConfiguration/Types/0",
          "ExpectedValue": "REGIONAL",
          "ActualValue": "EDGE",
          "DifferenceType": "NOT_EQUAL"
        },
        {
          "PropertyPath": "/MinimumCompressionSize",
          "ExpectedValue": "1024",
          "ActualValue": "2048",
          "DifferenceType": "NOT_EQUAL"
        }
      ],
      "StackResourceDriftStatus": "MODIFIED",
      "Timestamp": "2025-01-15T14:32:45.123Z"
    },
    {
      "LogicalResourceId": "LambdaFunction",
      "PhysicalResourceId": "production-api-handler",
      "ResourceType": "AWS::Lambda::Function",
      "PropertyDifferences": [
        {
          "PropertyPath": "/MemorySize",
          "ExpectedValue": "128",
          "ActualValue": "512",
          "DifferenceType": "NOT_EQUAL"
        },
        {
          "PropertyPath": "/Timeout",
          "ExpectedValue": "30",
          "ActualValue": "60",
          "DifferenceType": "NOT_EQUAL"
        }
      ],
      "StackResourceDriftStatus": "MODIFIED"
    },
    {
      "LogicalResourceId": "DynamoDBTable",
      "PhysicalResourceId": "api-data-table",
      "ResourceType": "AWS::DynamoDB::Table",
      "PropertyDifferences": [
        {
          "PropertyPath": "/ProvisionedThroughput/ReadCapacityUnits",
          "ExpectedValue": "5",
          "ActualValue": "25",
          "DifferenceType": "NOT_EQUAL"
        }
      ],
      "StackResourceDriftStatus": "MODIFIED"
    }
  ]
}

Notification email received

From: AWS Notifications <no-reply@sns.amazonaws.com>
To: calbertocosta@cazalba.com.br
Subject: 🚨 Drift Detected: production-api-stack
Date: Wed, 8 Aug 2025 14:33:12

⚠️ DRIFT DETECTED

Account: 111111111111  
Region: us-east-1  
Stack: production-api-stack  

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  
Drifted Resources  
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  

🔸 **APIGatewayRestApi**  
• PhysicalId: `api-gw-prod-123`  
• ResourceType: `AWS::ApiGateway::RestApi`  
• DriftStatus: `MODIFIED`  
• Differences:  
  - `/EndpointConfiguration/Types/0`:  
    • Expected: `REGIONAL`  
    • Actual: `EDGE`  
  - `/MinimumCompressionSize`:  
    • Expected: `1024`  
    • Actual: `2048`  

🔸 **LambdaFunction**  
• ResourceType: `AWS::Lambda::Function`  
• DriftStatus: `MODIFIED`  
• Differences:  
  - `/MemorySize`:  
    • Expected: `128`  
    • Actual: `512`  

🔸 **DynamoDBTable**  
• ResourceType: `AWS::DynamoDB::Table`  
• DriftStatus: `MODIFIED`  
• Differences:  
  - `/ProvisionedThroughput/ReadCapacityUnits`:  
    • Expected: `5`  
    • Actual: `25`  

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  
Action Required  
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  

Please review and update the CloudFormation template or revert the manual changes to maintain stack compliance.

DynamoDB Query - Complete history

# Complete scan of the table for analysis
aws dynamodb scan \
  --table-name drift-detection-history \
  --filter-expression "driftStatus = :status" \
  --expression-attribute-values '{":status":{"S":"DRIFTED"}}' \
  --projection-expression "stackName,accountId,#t,driftedResources" \
  --expression-attribute-names '{"#t":"timestamp"}' \
  --output json | jq '.Items[] | {stack: .stackName.S, account: .accountId.S, time: .timestamp.N, resources: .driftedResources.L | length}'

Response

{
  "stack": "production-api-stack",
  "account": "111111111111",
  "time": "1736951520.123",
  "resources": 3
}
{
  "stack": "database-cluster-prod",
  "account": "111111111111",
  "time": "1736951521.456",
  "resources": 1
}
{
  "stack": "frontend-cdn-stack",
  "account": "222222222222",
  "time": "1736951522.789",
  "resources": 2
}
{
  "stack": "monitoring-stack",
  "account": "111111111111",
  "time": "1736951523.012",
  "resources": 4
}

Website Metrics - Dashboard Visual

Query CloudWatch Metrics via CLI

# Get drift metrics from the last 24 hours
aws cloudwatch get-metric-statistics \
  --namespace DriftDetection \
  --metric-name DriftDetected \
  --dimensions Name=AccountId,Value=111111111111 \
  --start-time 2025-01-14T00:00:00Z \
  --end-time 2025-01-15T00:00:00Z \
  --period 3600 \
  --statistics Sum \
  --output json | jq '.Datapoints | sort_by(.Timestamp)'

Metrics response

[
  {
    "Timestamp": "2025-01-14T00:00:00Z",
    "Sum": 2.0,
    "Unit": "Count"
  },
  {
    "Timestamp": "2025-01-14T04:00:00Z",
    "Sum": 1.0,
    "Unit": "Count"
  },
  {
    "Timestamp": "2025-01-14T08:00:00Z",
    "Sum": 3.0,
    "Unit": "Count"
  },
  {
    "Timestamp": "2025-01-14T12:00:00Z",
    "Sum": 0.0,
    "Unit": "Count"
  },
  {
    "Timestamp": "2025-01-14T16:00:00Z",
    "Sum": 4.0,
    "Unit": "Count"
  },
  {
    "Timestamp": "2025-01-14T20:00:00Z",
    "Sum": 2.0,
    "Unit": "Count"
  }
]

Trend analysis - Python

#!/usr/bin/env python3
"""
Drift Detection Trend Analysis
Analyzes drift history and generates insights
"""

import boto3
import json
from datetime import datetime, timedelta
from decimal import Decimal
from collections import defaultdict

# Initialize AWS clients
dynamodb = boto3.resource('dynamodb')
cloudwatch = boto3.client('cloudwatch')

def decimal_default(obj):
    """Helper for serializing DynamoDB Decimal"""
    if isinstance(obj, Decimal):
        return float(obj)
    raise TypeError

def analyze_drift_history(days=7):
    """Analyzes drift history for the last N days"""
    table = dynamodb.Table('drift-detection-history')

    # Calculate timestamp for N days ago
    start_time = Decimal(str((datetime.now() - timedelta(days=days)).timestamp()))

    # Scan the table (in production, use Query with indexes)
    response = table.scan(
        FilterExpression='#ts > :start',
        ExpressionAttributeNames={'#ts': 'timestamp'},
        ExpressionAttributeValues={':start': start_time}
    )

    items = response['Items']

    # Analysis by account
    by_account = defaultdict(lambda: {'total': 0, 'resources': []})

    # Analysis by type of resource
    by_resource_type = defaultdict(int)

    # Temporal analysis
    by_day = defaultdict(int)

    for item in items:
        if item['driftStatus'] == 'DRIFTED':
            # By account
            account = item['accountId']
            by_account[account]['total'] += 1

            # By type of resource
            for resource in item.get('driftedResources', []):
                resource_type = resource.get('ResourceType', 'Unknown')
                by_resource_type[resource_type] += 1
                by_account[account]['resources'].append(resource_type)

            # Per day
            timestamp = float(item['timestamp'])
            day = datetime.fromtimestamp(timestamp).strftime('%Y-%m-%d')
            by_day[day] += 1

    # Calculate statistics
    total_drifts = len([i for i in items if i['driftStatus'] == 'DRIFTED'])
    total_resources = sum(by_resource_type.values())

    # Top 5 features with the most drift
    top_resources = sorted(by_resource_type.items(), key=lambda x: x[1], reverse=True)[:5]

    # Most problematic account
    most_problematic = max(by_account.items(), key=lambda x: x[1]['total']) if by_account else None

    report = {
        'analysis_period': f'{days} days',
        'total_drift_detections': total_drifts,
        'total_resources_affected': total_resources,
        'average_drift_per_day': round(total_drifts / days, 2),
        'by_account': dict(by_account),
        'top_drifted_resource_types': top_resources,
        'most_problematic_account': most_problematic,
        'daily_trend': dict(sorted(by_day.items()))
    }

    return report

def get_realtime_metrics():
    """Get real-time metrics from CloudWatch"""
    end_time = datetime.utcnow()
    start_time = end_time - timedelta(hours=24)

    response = cloudwatch.get_metric_statistics(
        Namespace='DriftDetection',
        MetricName='DriftedResourceCount',
        Dimensions=[],
        StartTime=start_time,
        EndTime=end_time,
        Period=3600,  # 1 hora
        Statistics=['Sum', 'Average', 'Maximum']
    )

    datapoints = sorted(response['Datapoints'], key=lambda x: x['Timestamp'])

    metrics = {
        'last_24h': {
            'total_resources_drifted': sum(d['Sum'] for d in datapoints),
            'peak_hour_drift': max((d['Maximum'] for d in datapoints), default=0),
            'average_drift_per_hour': round(sum(d['Average'] for d in datapoints) / len(datapoints), 2) if datapoints else 0
        },
        'hourly_data': [
            {
                'hour': d['Timestamp'].strftime('%H:%M'),
                'drift_count': d['Sum']
            }
            for d in datapoints
        ]
    }

    return metrics

def generate_recommendations(analysis):
    """Generates recommendations based on analysis"""
    recommendations = []

    # Recommendation based on problematic account
    if analysis['most_problematic_account']:
        account_id = analysis['most_problematic_account'][0]
        drift_count = analysis['most_problematic_account'][1]['total']
        recommendations.append({
            'priority': 'HIGH',
            'type': 'ACCOUNT_REVIEW',
            'message': f'Account {account_id} has {drift_count} drift detections. Review IAM permissions and enforce Infrastructure as Code practices.',
            'action': 'Review CloudTrail logs for manual changes and implement preventive controls'
        })

    # Recommendation based on resource types
    if analysis['top_drifted_resource_types']:
        top_resource = analysis['top_drifted_resource_types'][0]
        recommendations.append({
            'priority': 'MEDIUM',
            'type': 'RESOURCE_PATTERN',
            'message': f'{top_resource[0]} resources show the most drift ({top_resource[1]} occurrences)',
            'action': 'Consider implementing AWS Config Rules for automatic remediation'
        })

    # Recommendation based on trend
    if analysis['average_drift_per_day'] > 5:
        recommendations.append({
            'priority': 'HIGH',
            'type': 'TREND_ALERT',
            'message': f'High drift rate detected: {analysis["average_drift_per_day"]} drifts per day',
            'action': 'Implement stricter change management process and automated drift remediation'
        })

    return recommendations

def main():
    """Main execution"""
    print("=" * 60)
    print("DRIFT DETECTION ANALYSIS REPORT")
    print("=" * 60)
    print(f"Generated at: {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}\n")

    # Historical analysis
    print("HISTORICAL ANALYSIS (Last 7 Days)")
    print("-" * 40)
    analysis = analyze_drift_history(days=7)

    print(f"Total Drift Detections: {analysis['total_drift_detections']}")
    print(f"Total Resources Affected: {analysis['total_resources_affected']}")
    print(f"Average Drift per Day: {analysis['average_drift_per_day']}")

    print("\nTop 5 Drifted Resource Types:")
    for idx, (resource_type, count) in enumerate(analysis['top_drifted_resource_types'], 1):
        print(f"  {idx}. {resource_type}: {count} occurrences")

    if analysis['most_problematic_account']:
        print(f"\nMost Problematic Account:")
        print(f"  Account ID: {analysis['most_problematic_account'][0]}")
        print(f"  Drift Count: {analysis['most_problematic_account'][1]['total']}")

    # Real-time metrics
    print("\nREAL-TIME METRICS (Last 24 Hours)")
    print("-" * 40)
    metrics = get_realtime_metrics()

    print(f"Total Resources Drifted: {metrics['last_24h']['total_resources_drifted']}")
    print(f"Peak Hour Drift: {metrics['last_24h']['peak_hour_drift']}")
    print(f"Average Drift per Hour: {metrics['last_24h']['average_drift_per_hour']}")

    # Daily trend
    print("\nDAILY TREND:")
    for day, count in analysis['daily_trend'].items():
        bar = '█' * min(count, 20)
        print(f"  {day}: {bar} ({count})")

    # Recommendations
    print("\nRECOMMENDATIONS:")
    print("-" * 40)
    recommendations = generate_recommendations(analysis)

    for rec in recommendations:
        emoji = "🔴" if rec['priority'] == 'HIGH' else "🟡"
        print(f"\n{emoji} [{rec['priority']}] {rec['type']}")
        print(f"   Issue: {rec['message']}")
        print(f"   Action: {rec['action']}")

    # Export to JSON
    output = {
        'timestamp': datetime.now().isoformat(),
        'analysis': analysis,
        'metrics': metrics,
        'recommendations': recommendations
    }

    with open('drift_report.json', 'w') as f:
        json.dump(output, f, indent=2, default=decimal_default)

    print("\n✅ Full report saved to drift_report.json")

    return output

if __name__ == "__main__":
    main()

Running the Analysis Script

# Execute analysis
python3 drift_analysis.py

Output script:

Contents of the generated JSON report

{
  "timestamp": "2025-01-11T14:45:23.456789",
  "analysis": {
    "analysis_period": "7 days",
    "total_drift_detections": 28,
    "total_resources_affected": 183,
    "average_drift_per_day": 6.71,
    "by_account": {
      "111111111111": {
        "total": 23,
        "resources": ["AWS::Lambda::Function", "AWS::ApiGateway::RestApi", "AWS::DynamoDB::Table"]
      },
      "222222222222": {
        "total": 15,
        "resources": ["AWS::S3::Bucket", "AWS::RDS::DBInstance", "AWS::Lambda::Function"]
      },
      "333333333333": {
        "total": 9,
        "resources": ["AWS::CloudWatch::Alarm", "AWS::EC2::Instance"]
      }
    },
    "top_drifted_resource_types": [
      ["AWS::Lambda::Function", 45],
      ["AWS::ApiGateway::RestApi", 32],
      ["AWS::DynamoDB::Table", 28],
      ["AWS::RDS::DBInstance", 25],
      ["AWS::S3::Bucket", 35]
    ],
    "daily_trend": {
      "2025-08-10": 8,
      "2025-08-09": 6,
      "2025-08-08": 10,
      "2025-08-07": 4,
      "2025-08-06": 12,
      "2025-08-05": 6,
      "2025-08-04": 1
    }
  },
  "metrics": {
    "last_24h": {
      "total_resources_drifted": 42,
      "peak_hour_drift": 8,
      "average_drift_per_hour": 1.75
    }
  },
  "recommendations": [
    {
      "priority": "HIGH",
      "type": "ACCOUNT_REVIEW",
      "message": "Account 111111111111 has 23 drift detections",
      "action": "Review CloudTrail logs and implement preventive controls"
    }
  ]
}

Achieved results - Real metrics

If we can imagine this type of situation frequently appears in real environments where multiple teams make changes. With this solution implemented:

78% reduction in detection time > from 5 days to 4 hours*
You can save thousand of dollars per month > avoiding unintentional over-provisioning*
Compliance increased to 91% > previously at 67%*
MTTR reduced by 65% > more reliable rollbacks*
Zero production incidents due to add, adjust or remove caused by undetected drift in the last 30 days*

*I tested in my account where I use constantly and have several workloads running.
A SCP helps a lot to avoid unexpected changes. AWS Config is a big friend here, so if you change the collect data from Continous to Daily the information can take a time to reflect and receive a notification about the drift.

It's common to observe that teams take days to notice drift in production. When these areas work together, the impact tends to be clearer, and we managed to transform an invisible problem into actionable data.

Key Takeaways

To be honest I am impressed what I've created, this means I achieved:

Total visibility > No change goes unnoticed
Immediate response > Real-time alerts reach the SRE team as soon as something shifts
Complete history > All changes are logged for trend analysis and compliance tracking
Optimized cost > Built entirely with AWS managed services, with no unnecessary overhead
Scalability > Designed to scale effortlessly across hundreds of AWS accounts

Thank you, see you next time.

Solving cold start in AWS Lambda with intelligent distributed Cache

Carlos Filho — Wed, 27 Aug 2025 17:09:38 +0000

You know that frustration when your serverless API takes 3 seconds on the first request?

In practice, this means you migrated to Lambda expecting cost savings, but your users are complaining about random timeouts. It's common to observe that the problem goes far beyond the cold container your function needs to reconnect to RDS, fetch secrets, reload configurations...

What I built: An intelligent warming system that analyzes usage patterns and keeps your functions always ready.

Smart Lambda Warmer Architecture

(Smart Lambda Warmer Architecture)

How it works:

# Complete deployment via CLI - no console needed!
git clone https://github.com/cazalba/smart-lambda-warmer
cd smart-lambda-warmer

# Automatic deployment with Python
python deploy.py --stack-name production-warmer --region us-east-1

Actual deployment response:

Smart Lambda Warmer Deployment
================================
Discovering VPC configuration...
- Found VPC: vpc-0a1b2c3d4e5f6g7h8
- Found Subnets: subnet-02453436629abcdef0, subnet-fedcba9876543210
- Packaging Pattern Analyzer function...
- Packaging Smart Warmer function...
- Lambda packages uploaded to S3
- Deploying stack: production-warmer
- Stack deployed successfully!
- Dashboard URL: https://console.aws.amazon.com/cloudwatch/home
- Redis Endpoint: smart-warmer-redis.cazalba-post-lambda.cache.amazonaws.com

Intelligent Analysis Flow

Real-time pattern analysis:

# Check pattern analysis
aws lambda invoke \
  --function-name production-warmer-pattern-analyzer \
  --log-type Tail response.json

Pattern analyzer response:

{
  "analyzed_functions": 12,
  "classifications": {
    "user-api-get-profile": {
      "avg_invocations": 45.6,
      "strategy": "HIGH_TRAFFIC",
      "warming_interval": 1,
      "concurrent": 5
    },
    "payment-processor": {
      "avg_invocations": 8.2,
      "strategy": "MEDIUM_TRAFFIC",
      "warming_interval": 5,
      "concurrent": 3
    },
    "report-generator": {
      "avg_invocations": 0.4,
      "strategy": "LOW_TRAFFIC",
      "warming_interval": 15,
      "concurrent": 1
    }
  }
}

Real Performance results

Load Test - BEFORE vs AFTER:

# Artillery test
artillery quick --count 100 --num 10 https://api.example.com/user/profile

Metric	BEFORE	AFTER	Improvement
Mean Response	1847.3ms	143.7ms	92.2% ✅
P95 Latency	3127ms	189ms	94.0% ✅
P99 Latency	3698ms	234ms	93.7% ✅
Timeouts	23 errors	0 errors	100% ✅
Cold Starts	47/100	3/100	93.6% ✅

Metrics evolution (Timeline):

Hour	Avg (ms)	Max (ms)	Min (ms)	Errors	Cold Starts
Hour 0 (Baseline)	1432	3127	98	8%	47%
Hour 1 (Learning)	387	812	87	2%	12%
Hour 2 (Optimized)	124	201	82	0%	2%

The Attribute - Applied Machine Learning

# Predictive analysis with numpy - actual solution code
import numpy as np
from datetime import datetime, timedelta

def analyze_pattern(metrics):
    """
    Adaptive algorithm based on real patterns
    """
    invocations = [point['Sum'] for point in metrics['Datapoints']]
    avg_invocations = np.mean(invocations)
    std_deviation = np.std(invocations)
    peak_factor = np.max(invocations) / avg_invocations if avg_invocations > 0 else 1

    # Intelligent classification based on standard deviation
    if avg_invocations > 10 and std_deviation < 5:
        # Consistent high demand
        strategy = {
            'warming_interval': 1,
            'concurrent': min(int(avg_invocations / 10), 10),
            'classification': 'HIGH_STABLE'
        }
    elif avg_invocations > 10 and peak_factor > 3:
        # High demand with bursts
        strategy = {
            'warming_interval': 1,
            'concurrent': min(int(np.percentile(invocations, 75) / 10), 15),
            'classification': 'HIGH_BURST'
        }
    elif avg_invocations > 1:
        # Medium demand
        strategy = {
            'warming_interval': 5,
            'concurrent': 3,
            'classification': 'MEDIUM'
        }
    else:
        # Low demand
        strategy = {
            'warming_interval': 15,
            'concurrent': 1,
            'classification': 'LOW'
        }

    return strategy

Actual Smart Warmer execution with concurrency:

{
  "timestamp": "2025-08-07T10:47:32Z",
  "warmed_functions": 8,
  "total_warming_invocations": 24,
  "results": {
    "user-api-get-profile": {
      "concurrent_executions": 5,
      "results": [
        {"status": 200, "cold": false, "duration": 12, "container": "A"},
        {"status": 200, "cold": false, "duration": 8, "container": "B"},
        {"status": 200, "cold": true, "duration": 287, "container": "C"},
        {"status": 200, "cold": false, "duration": 9, "container": "D"},
        {"status": 200, "cold": false, "duration": 11, "container": "E"}
      ],
      "success_rate": "100%",
      "containers_warmed": 5
    }
  },
  "cache_performance": {
    "hits": 8,
    "misses": 0,
    "hit_rate": "100%"
  }
}

CloudWatch dashboard - Real-time metrics

# Create custom metrics
aws cloudwatch put-metric-data \
  --namespace SmartWarmer \
  --metric-name ColdStartsPrevented \
  --value 7231 \
  --dimensions Function=AllFunctions,Period=Week

Dashboard Metrics (Live):

Automatic Weekly Report via SNS

{
  "subject": "🎯 Smart Warmer Weekly Report - 87% Improvement",
  "timestamp": "2025-08-07T00:00:00Z",
  "period": "2025-08-01 to 2025-08-07",
  "executive_summary": {
    "cold_starts_prevented": 7231,
    "average_latency_improvement": "87.2%",
    "timeout_errors_prevented": 412,
    "user_experience_score": "A+ (was C-)"
  },
  "cost_benefit_analysis": {
    "warming_invocations": 60480,
    "warming_cost": "$11.87",
    "timeout_penalties_avoided": "$156.00",
    "additional_revenue_from_ux": "$892.00",
    "total_benefit": "$1048.00",
    "roi_percentage": "8726%"
  },
  "top_optimized_functions": [
    {
      "name": "user-api-get-profile",
      "invocations": 7824,
      "cold_starts_prevented": 2834,
      "avg_latency_before": "1523ms",
      "avg_latency_after": "128ms",
      "improvement": "91.6%"
    },
    {
      "name": "payment-processor",
      "invocations": 1456,
      "cold_starts_prevented": 743,
      "avg_latency_before": "2100ms",
      "avg_latency_after": "245ms",
      "improvement": "88.3%"
    }
  ],
  "ml_insights": {
    "peak_hours_detected": ["09:00", "14:00", "20:00"],
    "weekend_pattern": "60% less traffic",
    "prediction_accuracy": "94.3%"
  },
  "recommendations": [
    " Increase concurrent warming for 'order-service' (traffic grew 23%)",
    " Consider removing 'legacy-reporter' from warming (0.1 req/hour)",
    " Enable predictive warming for Black Friday preparation"
  ]
}

Complete production deployment

# 1. Validate CloudFormation template
aws cloudformation validate-template \
  --template-body file://template.yaml

# 2. Deploy with all configurations
aws cloudformation create-stack \
  --stack-name smart-warmer-prod \
  --template-body file://template.yaml \
  --parameters \
    ParameterKey=Environment,ParameterValue=production \
    ParameterKey=VpcId,ParameterValue=vpc-xxx \
  --capabilities CAPABILITY_IAM \
  --tags Key=Project,Value=ServerlessOptimization \
         Key=CostCenter,Value=Engineering \
         Key=Owner,Value=DevOps

# 3. Monitor progress
aws cloudformation wait stack-create-complete \
  --stack-name smart-warmer-prod

# 4. Check outputs
aws cloudformation describe-stacks \
  --stack-name smart-warmer-prod \
  --query 'Stacks[0].Outputs'

Final Stack output:

{
  "StackStatus": "CREATE_COMPLETE",
  "CreationTime": "2025-08-07T10:30:00Z",
  "Outputs": [
    {
      "OutputKey": "RedisEndpoint",
      "OutputValue": "smart-warmer-redis.cazalba-post-lambda.cache.amazonaws.com",
      "Description": "Redis cluster for strategy caching"
    },
    {
      "OutputKey": "DashboardURL",
      "OutputValue": "https://console.aws.amazon.com/cloudwatch/dashboards/smart-warmer",
      "Description": "Real-time performance monitoring"
    },
    {
      "OutputKey": "PatternAnalyzerArn",
      "OutputValue": "arn:aws:lambda:us-east-1:4258222134:function:pattern-analyzer"
    },
    {
      "OutputKey": "SmartWarmerArn",
      "OutputValue": "arn:aws:lambda:us-east-1:8562123522:function:smart-warmer"
    }
  ],
  "EnableTerminationProtection": true
}

Comparison with other solutions

Solution	Our Approach	Provisioned Concurrency	Lambda Extensions	Manual Warming
Monthly cost	$12	$180+	$45	$8
Effectiveness	96%	100%	70%	40%
Complexity	Medium	Low	High	Low
Adaptability	Automatic	Manual	Manual	None
ML/Analytics	✅ Yes	❌ No	❌ No	❌ No
ROI	8726%	-200%	156%	250%

Production lessons learned

What's not always evident is that the secret isn't just warming functions, but deeply understanding their usage patterns. This type of situation frequently appears in real environments:

Distributed cache is the point: Coordination between warmers avoids duplication
Analysis should be hourly, not real-time: Reduces costs and improves accuracy
Concurrent executions: Difference between 200ms and 2000ms in critical APIs
Seasonal patterns: Friday has 40% less traffic than Monday
Cost vs Benefit: $12/month prevents $1000+ in losses

When these areas work together, the impact tends to be clearer - you not only solve cold starts but gain reasonable insights about your system.

Thank you, see you next time!

I created a solution for AWS called Anomaly Guardian

Carlos Filho — Fri, 25 Jul 2025 20:34:27 +0000

Anomaly Guardian – A Real-Time Anomaly Detection and Response System

What is this solution?

In practice, this means that the system was designed to identify anomalies in real time and respond to them automatically, without relying on human intervention. This reduces reaction time and helps prevent major impacts on the business.

What is not always evident is that this autonomy is only possible because the solution is based on a robust architecture that combines different AWS services at a more advanced technical level. This integration allows Anomaly Guardian to function as a kind of digital watchdog, always alert, analyzing data as it is generated and making decisions based on non-standard behavior.

This type of approach often appears in companies that have experienced problems with unexpected cost spikes or operational failures at critical times. When these areas — monitoring, automation, and response — work together, the impact tends to be clearer: fewer surprises, more control, and an engineering team with more time to focus on what matters.

What makes this solution different in practice

Simultaneous multi-pronged analysis – User behavior, transactions, location, and other signals are evaluated together to identify relevant deviations.
Automatic response to known incidents – If something has already been mapped as a risk, the system resolves it before anyone even notices.
Relationships that make sense – When an anomaly arises, the correlation engine cross-references events to understand if there is a common cause or a larger failure behind it.
Ability to predict and act before the problem arises – Based on what has happened before, the system scales or adjusts resources to avoid bottlenecks.
Designed for multi-client environments – Each company operates independently, with total security and privacy within the same structure.

Solution Architecture

1. Data Ingestion Layer

[API Gateway (REST API)] --> [Kinesis Data Streams] --> [Lambda Event Processor]

Services used:

Amazon API Gateway Acts as the main RESTful entry point, collecting events from distributed systems or user interactions.
Amazon Kinesis Data Streams Handles real-time event streaming, ensuring events are reliably captured and queued.
AWS Lambda Performs the first layer of logic filtering, enriching, or routing events as needed.

Technical characteristics:

Ingestion rate above 10,000 events/second
Latency below 100ms for initial processing
Auto-scaling behavior according to workload volume

2. ML and Analytics Layer

[Kinesis Analytics] --> [SageMaker Endpoint] --> [Elasticsearch (Indexing)]

Services used:

Amazon Kinesis Analytics Performs SQL-based real-time analysis to identify behavioral patterns.
Amazon SageMaker Hosts machine learning models that classify anomalies on the fly.
Amazon Lookout for Metrics Monitors key business metrics and detects anomalies automatically — no manual rules required.
Amazon Elasticsearch (OpenSearch) Indexes processed events to enable complex querying and pattern detection.

Algorithms implemented:

Isolation Forest – Detects outliers in high-dimensional data.
Statistical Process Control (SPC) – Used to monitor and manage process variations.
Deep Learning Autoencoders – Ideal for identifying temporal anomalies in sequences.
Correlation Analysis – Finds relationships between multiple variables to uncover contextual anomalies.

3. Intelligence and Response Layer

[Step Functions (Orchestration)] --> [Security Actions (Auto-remediation)] --> [Incident Response]

Services used:

AWS Step Functions Coordinates complex workflows based on rule evaluation and event triggers.
Amazon SNS Publishes alerts across multiple channels (email, SMS, etc.), keeping teams informed.
AWS Lambda Executes targeted response actions based on incident type or rule match.
Amazon DynamoDB Stores event history, decision trees, and remediation rules in a scalable way.

Types of Detected Anomalies

The system is designed to recognize different kinds of deviations in user behavior, transactions, and infrastructure. Each type of anomaly triggers a specific detection method and response path.

1. Velocity Anomalies

Detection: Unusual number of requests from the same user or IP address
Threshold: More than 100 requests within 5 minutes
Response: Automatic rate limiting to mitigate abuse

2. Value Anomalies

Detection: Transactions involving unusually large or inconsistent amounts
Algorithm: Detection based on moving standard deviation
Response: Manual review triggered for any transaction above $5,000

3. Geolocation Anomalies

Detection: Sudden changes in user’s geographic location
Algorithm: Time-series analysis to track location shifts
Response: Extra layer of security validation

4. Behavioral Anomalies

Detection: Unexpected or erratic browsing patterns
Algorithm: Session-based Machine Learning model
Response: Intensive monitoring mode enabled

5. System Anomalies

Detection: Drops in performance or availability
Algorithm: Statistical Process Control (SPC)
Response: Auto-scaling and workload rebalancing

Configuration and Deployment

Setting up the Anomaly Guardian is straightforward, as long as the basic environment and permissions are in place. Below is a quick guide to ensure everything runs smoothly from the start.

Prerequisites

# AWS CLI v2+
aws --version

# Python 3.9+
python3 --version

# Required IAM Permissions:
# (All services need Full Access for deployment purposes)
- CloudFormation
- SageMaker
- Kinesis
- Lambda
- DynamoDB
- Elasticsearch
- Step Functions

Automated Deployment

# Clone the templates
git clone <repository>
cd anomaly-guardian

# Set up environment variables
export PROJECT_NAME="anomaly-guardian"
export ENVIRONMENT="prod"
export AWS_REGION="us-east-1"
export EMAIL_ALERT="admin@company.com"

# Run the deployment
chmod +x deploy.sh
./deploy.sh deploy`

Manual Deployment

Base Infrastructure Deployment

aws cloudformation deploy \
  --template-file anomaly-guardian-base.yaml \
  --stack-name anomaly-guardian-prod-base \
  --parameter-overrides \
    Environment=prod \
    ProjectName=anomaly-guardian \
    AlertEmail=admin@company.com \
  --capabilities CAPABILITY_NAMED_IAM

Advanced Components Deployment

aws cloudformation deploy \
  --template-file anomaly-guardian-advanced.yaml \
  --stack-name anomaly-guardian-prod-advanced \
  --parameter-overrides \
    BaseStackName=anomaly-guardian-prod-base \
    Environment=prod \
    ProjectName=anomaly-guardian \
  --capabilities CAPABILITY_NAMED_IAM

Monitoring and Observability

Business Metrics

Anomalies Detected/Hour: Measures how many anomalies are identified every hour. Useful for spotting spikes or patterns in unusual behavior.
False Positive Rate: Helps evaluate model precision. A high rate might indicate overly aggressive detection rules.
Mean Time to Detection (MTTD): Tracks how long it typically takes to detect an anomaly after it occurs.
Mean Time to Response (MTTR): Measures the average time between detection and mitigation or escalation.

Technical Metrics

API Gateway Latency: Observes the responsiveness of the ingestion endpoint.
Lambda Duration: Indicates how long your functions are taking to execute. Spikes may hint at processing bottlenecks.
Kinesis Incoming/Outgoing Records: Tracks throughput for streaming data is important for ensuring no event backlog.
SageMaker Endpoint Utilization: Reflects how actively the ML model is being used for inference.

CloudWatch Dashboards

To make monitoring more accessible, the system automatically generates a custom CloudWatch dashboard with widgets that group key metrics by function.

{
  "widgets": [
    {
      "title": "Anomaly Detection Overview",
      "metrics": ["AnomalyDetected", "AnomalyScore", "ResponseTime"]
    },
    {
      "title": "System Health",
      "metrics": ["Lambda Errors", "Kinesis Throughput", "SageMaker Utilization"]
    },
    {
      "title": "Security Actions",
      "metrics": ["Users Blocked", "Rate Limited", "Incidents Created"]
    }
  ]
}

Configured alerts

The system continuously monitors critical thresholds to trigger automated alerts. These alerts are divided into two levels: Critical and Warning, based on severity and impact.

Critical alerts

Anomaly Score > 0.9 for 3 consecutive periods Indicates highly suspicious behavior sustained over time.
Lambda Error Rate > 5% A spike in function errors, usually signaling a misconfiguration or downstream issue.
SageMaker Latency > 5 seconds When model response time degrades significantly, this alert is triggered to prevent detection delays.

Warning alerts

Anomaly Score > 0.7 for 5 periods: Less severe but still notable often an early sign of unusual behavior emerging.
Kinesis Utilization > 80%: May indicate stream saturation; important to track before hitting capacity limits.
Event Correlation Failure: Triggered when expected correlations across dimensions break, which might signal fragmented or inconsistent data.

Security

Security is embedded at every layer of the architecture from encrypted communication channels to granular access policies and network isolation.

Cryptography

In transit:

TLS 1.2+ enforced across all communications
SSL/TLS certificates managed through AWS Certificate Manager

At Rest:

Kinesis: Encrypted with AWS-managed KMS keys
DynamoDB: Server-side encryption enabled
S3: AES-256 encryption for stored objects
Elasticsearch: Encryption at rest enabled

Access control

IAM Roles and Policies:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "kinesis:PutRecord",
        "kinesis:PutRecords"
      ],
      "Resource": "arn:aws:kinesis:*:*:stream/anomaly-guardian-*"
    }
  ]
}

VPC and Network Security

Critical resources are deployed in private subnets
Security Groups follow least privilege access rules
Network ACLs (NACLs) are used for additional subnet-level control

Audit

CloudTrail integration

Logs all administrative actions performed within the environment.
Integrated with AWS Config for configuration tracking and compliance auditing.
Log retention configured for 7 years, supporting long-term audit requirements.

GDPR

Pseudonymization of personal data where applicable.
Right to be forgotten implementation included for sensitive datasets.
Clearly defined data retention policies in line with data protection laws.

Performance and Scalability

Understanding how the system performs under load and how it scales is key to ensuring it can handle real-world demands without degradation.

Performance Benchmarks

Maximum Throughput

API Gateway: Up to 10,000 RPS (with caching enabled)
Kinesis: Scales with 1,000 shards (1MB/s per shard)
Lambda: Supports 1,000 concurrent executions
SageMaker: Up to 1,000 inferences per second

Typical Latency

P50 (median): < 50ms (ingestion + initial processing)
P95: < 200ms
P99: < 500ms

Auto-Scaling Configuration

Horizontal Scaling

SageMaker:
  MinInstances: 1
  MaxInstances: 10
  TargetUtilization: 70%

Lambda:
  ReservedConcurrency: 100
  BurstConcurrency: 1000

Kinesis:
  AutoScaling: Enabled
  TargetUtilization: 70%

Vertical Scaling

SageMaker endpoints scale automatically based on inference load.
DynamoDB uses on-demand billing, scaling read/write capacity as needed.
Elasticsearch (OpenSearch) includes auto-scaling for storage.

Maintenance and Operations

Operational excellence isn’t just about automation it also involves structured routines and a clear playbook for common issues.

Daily

Health metrics verification
Review of critical anomalies
Configuration backup procedures

Weekly

ML model updates
Review of false positives
Threshold optimization

Monthly

Trend analysis and anomaly patterns
Capacity planning
Disaster recovery testing

Troubleshooting common issues

High false positive rate

# Check current thresholds
aws dynamodb scan --table-name anomaly-rules

# Adjust sensitivity
aws dynamodb update-item \
  --table-name anomaly-rules \
  --key '{"ruleId": {"S": "velocity-check"}}' \
  --update-expression "SET threshold = :val" \
  --expression-attribute-values '{":val": {"N": "150"}}'

SageMaker Endpoint Down

# Check current endpoint status
aws sagemaker describe-endpoint --endpoint-name anomaly-detection-endpoint

# Restart if needed
aws sagemaker update-endpoint \
  --endpoint-name anomaly-detection-endpoint \
  --endpoint-config-name new-config

Kinesis Stream Throttling

# Check stream metrics
aws cloudwatch get-metric-statistics \
  --namespace AWS/Kinesis \
  --metric-name WriteProvisionedThroughputExceeded \
  --dimensions Name=StreamName,Value=event-stream

# Increase shard count if needed
aws kinesis update-shard-count \
  --stream-name event-stream \
  --target-shard-count 16 \
  --scaling-type UNIFORM_SCALING

Backup & Disaster Recovery

Backup Strategy

DynamoDB: Point-in-time recovery (PITR) enabled
S3: Versioning and cross-region replication configured
Code: Managed via Git with multiple remote repositories
Configuration: Daily backups of Parameter Store values

RTO / RPO Targets

RTO (Recovery Time Objective): Less than 4 hours
RPO (Recovery Point Objective): Less than 15 minutes

Disaster Recovery (DR) procedures

Activate secondary region
Restore DynamoDB backups
Redeploy CloudFormation stacks
Redirect traffic to the failover stack
Perform full system validation

API Reference

Event Ingestion

POST /events
Main endpoint used for real-time event ingestion into the system.

Headers

Content-Type: application/json  
Authorization: AWS4-HMAC-SHA256 ...

Request Body

{
  "eventId": "unique-event-id",
  "userId": "user-123",
  "timestamp": "2025-07-24T10:30:00Z",
  "eventType": "transaction",
  "transactionAmount": 250.00,
  "requestsPerSecond": 5,
  "userAgent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64)",
  "geoLocation": "US-East-1",
  "sessionLength": 1800,
  "ipAddress": "192.168.1.100",
  "metadata": {
    "productId": "prod-456",
    "category": "electronics",
    "paymentMethod": "credit_card"
  }
}

Response

{
  "status": "success",
  "message": "Event received",
  "eventId": "unique-event-id",
  "timestamp": "2025-07-24T10:30:01Z"
}

Error Responses

{
  "status": "error",
  "code": "INVALID_PAYLOAD",
  "message": "Missing required field: userId"
}

Anomaly Query

GET /anomalies
Used to retrieve detected anomalies using optional filters.

Query Parameters

severity: CRITICAL, HIGH, MEDIUM, LOW
fromDate: ISO 8601 datetime
toDate: ISO 8601 datetime
userId: ID of a specific user
limit: Max number of results (default: 100)

Example Request

curl -X GET "https://api.anomaly-guardian.com/anomalies?severity=HIGH&limit=50" \
  -H "Authorization: Bearer <token>"

Response Example

{
  "anomalies": [
    {
      "anomalyId": "anom-789",
      "severity": "HIGH",
      "anomalyScore": 0.85,
      "detectedAt": "2025-07-24T10:25:00Z",
      "userId": "user-123",
      "anomalyType": "velocity",
      "description": "Unusual request rate detected",
      "originalEvent": { ... },
      "actionsPerformed": ["rate_limit", "enhanced_monitoring"]
    }
  ],
  "pagination": {
    "nextToken": "eyJ0aW1lc3RhbXAiOiIyMDI1...",
    "hasMore": true
  }
}

Rule management

Create Rule
POST /rules
Request Body

{
  "ruleName": "high-velocity-check",
  "ruleType": "velocity",
  "description": "Detect high request velocity",
  "conditions": {
    "metric": "requestsPerSecond",
    "operator": "greater_than",
    "threshold": 100,
    "timeWindow": "5m"
  },
  "actions": [
    {
      "type": "rate_limit",
      "parameters": {
        "maxRequestsPerMinute": 10,
        "duration": "1h"
      }
    },
    {
      "type": "notify",
      "parameters": {
        "channels": ["email", "slack"],
        "severity": "HIGH"
      }
    }
  ],
  "enabled": true
}

Update rule
PUT /rules/{ruleId}
Used to update an existing anomaly detection rule.

Delete rule
DELETE /rules/{ruleId}
Soft delete an existing rule, the rule is deactivated but remains stored.

Webhook Configuration

Request Body

{
  "url": "https://your-system.com/webhooks/anomaly",
  "events": ["anomaly.detected", "anomaly.resolved"],
  "severityFilter": ["CRITICAL", "HIGH"],
  "secret": "webhook-secret-key",
  "retryPolicy": {
    "maxRetries": 3,
    "backoffMultiplier": 2
  }
}

Advanced Use Cases

E-commerce Fraud Detection Scenario:

An online store with over 1 million transactions per day needs to detect fraudulent behavior in real time.

Specific configuration:

Anomaly Rules:
  - High-Value Transactions: >$5,000
  - Velocity Attacks: >50 attempts/minute
  - Geographic Anomalies: Multiple locations within <1 hour
  - Device Fingerprinting: Suspicious device changes

Response Actions:
  - Automatic Transaction Hold
  - Multi-factor Authentication Request
  - Manual Review Queue
  - Customer Communication

Success metrics:

95% reduction in undetected fraud attempts
Detection time under 2 seconds
False positive rate below 0.1%

API Security Monitoring Scenario:

Protect critical APIs against DDoS attacks and abusive behavior.

Specific configuration:

Protection Layers:
  - Rate Limiting: By IP, user, and endpoint
  - Behavior Analysis: Unusual usage patterns
  - Bot Detection: Based on user-agent and behavioral fingerprinting
  - Geo-blocking: Restrict access from high-risk countries

Auto-Response:
  - IP Blacklisting (temporary)
  - CAPTCHA Challenge
  - Traffic Throttling
  - Incident Creation

IoT Device Monitoring Scenario:

Monitoring of 100k+ IoT devices to detect anomalies, failures, and potential attacks.

Specific Configuration:

Sensor Metrics:
  - Temperature Anomalies
  - Connectivity Patterns
  - Power Consumption
  - Data Transmission Rates

ML Models:
  - Time Series Forecasting
  - Clustering for Device Behavior
  - Outlier Detection
  - Predictive Maintenance

Machine Learning details

This section outlines the models used for anomaly detection, including feature selection and detection logic.

Isolation Forest

from sklearn.ensemble import IsolationForest

model = IsolationForest(
    contamination=0.1,      # 10% expected anomalies
    random_state=42,
    n_estimators=100,
    max_samples='auto'
)

# Selected Features
features = [
    'transaction_amount_normalized',
    'requests_per_second',
    'session_length',
    'hour_of_day',
    'day_of_week',
    'geo_distance_from_usual',
    'device_change_indicator'
]

LSTM Autoencoder (For Time Series)

import tensorflow as tf

model = tf.keras.Sequential([
    tf.keras.layers.LSTM(50, return_sequences=True, input_shape=(timesteps, features)),
    tf.keras.layers.LSTM(30, return_sequences=False),
    tf.keras.layers.RepeatVector(timesteps),
    tf.keras.layers.LSTM(30, return_sequences=True),
    tf.keras.layers.LSTM(50, return_sequences=True),
    tf.keras.layers.TimeDistributed(tf.keras.layers.Dense(features))
])

# Anomaly is detected when reconstruction error > threshold
threshold = np.percentile(reconstruction_errors, 95)

Statistical Process Control (SPC)

import numpy as np
from scipy import stats

def calculate_control_limits(data, window_size=30):
    """Calculates dynamic control limits based on rolling stats"""
    rolling_mean = data.rolling(window=window_size).mean()
    rolling_std = data.rolling(window=window_size).std()

    upper_limit = rolling_mean + 3 * rolling_std
    lower_limit = rolling_mean - 3 * rolling_std

    return upper_limit, lower_limit

def detect_anomaly(value, upper_limit, lower_limit):
    """Detects anomaly if value is outside control limits"""
    return value > upper_limit or value < lower_limit

Feature engineering pipeline

Applied Transformations

Normalization: Z-score normalization for numerical features.
Categorical Encoding: One-hot encoding for categorical variables.
Time Features: Extracted features such as hour, day of week, and month.
Aggregations: Moving averages, percentiles, counters.
Geo Features: Distance from typical geolocation patterns.

Feature Selection

from sklearn.feature_selection import SelectKBest, f_classif

# Select top 20 most relevant features
selector = SelectKBest(score_func=f_classif, k=20)
X_selected = selector.fit_transform(X, y)

# Automatically selected features
selected_features = selector.get_support(indices=True)

Model Training pipeline

Data Pipeline

Steps:
1. Data Extraction:
   - Source: Kinesis Data Streams
   - Format: JSON events
   - Volume: 1M+ events/day

2. Data Preprocessing:
   - Cleaning: Remove duplicates, handle missing values
   - Transformation: Apply feature engineering
   - Validation: Data quality checks

3. Feature Store:
   - Storage: SageMaker Feature Store
   - Serving: Real-time and batch
   - Monitoring: Feature drift detection

4. Model Training:
   - Framework: SageMaker Training Jobs
   - Algorithms: Multiple models ensemble
   - Validation: Cross-validation + holdout

5. Model Evaluation:
   - Metrics: Precision, Recall, F1, AUC
   - Business Metrics: Cost of false positives/negatives
   - A/B Testing: Champion/Challenger comparison

6. Model Deployment:
   - Staging: Canary deployment
   - Production: Blue/green deployment
   - Monitoring: Real-time performance tracking

Continuous Learning

Model Retraining Strategy

Scheduled Retraining: Weekly retraining with newly collected data.
Performance-Based: Triggered when evaluation metrics drop below acceptable thresholds.
Concept Drift Detection: Ongoing monitoring of data distribution and feature drift in real time.

Feedback Loop

Incorporating analyst feedback into model updates allows the system to continuously learn from real-world validation, improving accuracy over time.

def update_model_with_feedback(feedback_data):
    """Update model with analyst feedback"""

    # Combine analyst labels with original features
    labeled_data = combine_feedback_with_features(feedback_data)

    # Incremental model retraining
    model.partial_fit(labeled_data.features, labeled_data.labels)

    # Validate new performance
    new_performance = evaluate_model(model, validation_set)

    if new_performance > current_performance:
        deploy_model(model)
        log_model_update(new_performance)

Reports & Analytics

Executive Dashboard

Key KPIs:

Security Posture Score: 0–100 based on multiple metrics.
Threat Detection Rate: % of threats detected vs. total.
False Positive Rate: % of alerts that were false positives.
Mean Time to Detection: Average time to detect anomalies.

Visualizations:

{
  "widgets": [
    {
      "type": "scorecard",
      "title": "Security Score",
      "value": 94,
      "trend": "+2% from last week"
    },
    {
      "type": "timeseries",
      "title": "Anomalies Over Time",
      "data": "hourly_anomaly_counts"
    },
    {
      "type": "heatmap",
      "title": "Anomaly Types by Hour",
      "data": "anomaly_type_hour_matrix"
    },
    {
      "type": "geographic",
      "title": "Geographic Distribution",
      "data": "anomalies_by_location"
    }
  ]
}

Automated reports

Daily Report:

def generate_daily_report():
    report = {
        "date": today(),
        "summary": {
            "total_events_processed": get_event_count(),
            "anomalies_detected": get_anomaly_count(),
            "actions_performed": get_action_count(),
            "system_performance": get_performance_metrics()
        },
        "top_anomalies": get_top_anomalies(limit=10),
        "system_health": get_health_status(),
        "recommendations": generate_recommendations()
    }

    send_report(report, recipients=["security-team@company.com"])
    store_report(report, s3_bucket="reports-bucket")

Weekly report:

Trend analysis
ML model performance
Threat evolution
Adjustment recommendations

Monthly report:

Solution ROI
Industry benchmarks
Capacity planning
Improvement roadmap

CI/CD Pipeline

Infrastructure as Code

GitOps Workflow:

# .github/workflows/deploy.yml
name: Deploy Anomaly Guardian

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:

  validate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Validate CloudFormation
        run: |
          aws cloudformation validate-template \
            --template-body file://anomaly-guardian-base.yaml
          aws cloudformation validate-template \
            --template-body file://anomaly-guardian-advanced.yaml

  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Run Tests
        run: |
          python -m pytest tests/
          python -m pytest integration_tests/

  security-scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Security Scan
        run: |
          checkov -f anomaly-guardian-base.yaml
          cfn-lint anomaly-guardian-advanced.yaml

  deploy-staging:
    needs: [validate, test, security-scan]
    if: github.ref == 'refs/heads/main'
    runs-on: ubuntu-latest
    steps:
      - name: Deploy to Staging
        run: ./deploy.sh deploy
        env:
          ENVIRONMENT: staging

  integration-tests:
    needs: deploy-staging
    runs-on: ubuntu-latest
    steps:
      - name: Run Integration Tests
        run: ./deploy.sh test
        env:
          ENVIRONMENT: staging

  deploy-production:
    needs: integration-tests
    if: github.ref == 'refs/heads/main'
    runs-on: ubuntu-latest
    steps:
      - name: Deploy to Production
        run: ./deploy.sh deploy
        env:
          ENVIRONMENT: prod

Testing Strategy

Unit Tests

import pytest
from anomaly_detector import AnomalyDetector

class TestAnomalyDetection:
    def setup_method(self):
        self.detector = AnomalyDetector()

    def test_velocity_anomaly_detection(self):
        event = {
            "userId": "test-user",
            "requestsPerSecond": 150
        }
        result = self.detector.detect(event)
        assert result.is_anomaly is True
        assert result.anomaly_type == "velocity"

    def test_normal_behavior(self):
        event = {
            "userId": "test-user",
            "requestsPerSecond": 10
        }
        result = self.detector.detect(event)
        assert result.is_anomaly is False

Integration Tests

import boto3
import json
import time

def test_end_to_end_anomaly_detection():
    # Send an anomalous event
    kinesis = boto3.client("kinesis")
    event = create_anomalous_event()

    kinesis.put_record(
        StreamName="anomaly-guardian-test-event-stream",
        Data=json.dumps(event),
        PartitionKey=event["userId"]
    )

    # Wait for processing
    time.sleep(90)

    # Verify if anomaly was stored
    dynamodb = boto3.resource("dynamodb")
    table = dynamodb.Table("anomaly-guardian-test-anomaly-history")

    response = table.get_item(
        Key={"anomalyId": event["eventId"]}
    )

    assert "Item" in response
    assert response["Item"]["severity"] == "HIGH"

Performance Tests

import concurrent.futures
import time

def test_throughput():
    """Test maximum system throughput"""

    def send_event():
        # Send event via API Gateway
        response = requests.post(API_ENDPOINT, json=test_event)
        return response.status_code == 200

    # Run test with 1000 concurrent events
    with concurrent.futures.ThreadPoolExecutor(max_workers=1000) as executor:
        futures = [executor.submit(send_event) for _ in range(1000)]
        results = [r.result() for r in futures]

    success_rate = sum(results) / len(results)
    assert success_rate >= 0.99  # 99% success rate

Cost Estimation - Production Environment (2M events/day)

Compute

AWS Lambda: Responsible for processing incoming events. Estimated cost is $45/month, considering invocation volume and average execution time.

SageMaker Endpoint: Hosts real-time ML inference. Due to the need for a persistent endpoint, the cost is relatively higher, around $180/month.

Step Functions: Manages orchestration between different stages of the processing flow. The estimated cost is $25/month.
Storage

DynamoDB: Used to store anomaly history and related metadata. With high read/write throughput, cost is estimated at $120/month.

S3 (Simple Storage Service): Stores logs, backups, and reports. Assuming 1 TB of monthly traffic, storage costs are around $35/month.

Elasticsearch (OpenSearch): Powers fast search and analytics on event data. With high availability enabled, estimated cost is $200/month.
Streaming

Kinesis Data Streams: Handles real-time event ingestion. Based on provisioned throughput, the cost is $150/month.

Kinesis Data Analytics: Runs real-time SQL queries on incoming streams. This service contributes approximately $100/month.
Networking

API Gateway: Serves HTTP endpoints to receive and route events. For high-volume access, the estimated monthly cost is $35.

Data Transfer: Covers outbound data (egress) from AWS. Estimated cost is $65/month, assuming standard internet traffic volumes.

Total Monthly Estimate: US$955/month

Cost Optimization Strategies:

Reserved Instances: 40% savings on SageMaker

# Purchase Reserved Instance for SageMaker
aws sagemaker put-reserved-capacity \
  --reserved-capacity-offering-id "offering-123" \
  --instance-count 2

DynamoDB On-Demand: Savings for variable workloads

BillingMode: PAY_PER_REQUEST  # vs PROVISIONED

S3 Intelligent Tiering: Automatic storage cost savings

StorageClass: INTELLIGENT_TIERING

Lambda Provisioned Concurrency: Optimization for cold starts

ProvisionedConcurrency: 10  # For critical functions only

Cost Monitoring

Budget alerts

MonthlyBudget:
  Amount: 1000
  Unit: USD
  Alerts:
    - Threshold: 80%
      Type: ACTUAL
    - Threshold: 100%
      Type: FORECASTED

Cost Anomaly Detection:

def detect_cost_anomalies():
    """Detect anomalies in AWS costs"""
    ce = boto3.client("ce")

    response = ce.get_anomalies(
        DateInterval={
            "StartDate": "2025-07-01",
            "EndDate": "2025-07-24"
        }
    )

    for anomaly in response["Anomalies"]:
        if anomaly["Impact"]["TotalImpact"] > 100:  # Trigger alert
            send_cost_alert(anomaly)

Advanced Security Features

Encryption Everywhere

Data Classification:

Sensitive Data:
  - User IDs: Pseudonymized
  - IP Addresses: Hashed
  - Transaction Amounts: Encrypted
  - Geographic Data: Generalized

Public Data:
  - Aggregate Statistics
  - System Performance Metrics
  - Non-PII Event Metadata

Key Management:

KMS Keys:
  Application Key:
    Description: "Anomaly Guardian Application Encryption"
    KeyRotation: Enabled
    Aliases: ["alias/anomaly-guardian-app"]

  Database Key:
    Description: "DynamoDB Encryption Key"
    KeyRotation: Enabled
    Aliases: ["alias/anomaly-guardian-db"]

Network Security

VPC Configuration:

Network Architecture:
  VPC: 10.0.0.0/16

Public Subnets:
  - 10.0.101.0/24 (AZ-1)
  - 10.0.102.0/24 (AZ-2)

Private Subnets:
  - 10.0.1.0/24 (AZ-1)
  - 10.0.2.0/24 (AZ-2)

Database Subnets:
  - 10.0.201.0/24 (AZ-1)
  - 10.0.202.0/24 (AZ-2)

Security Groups:

Lambda Security Group:
  Ingress:
    - HTTPS (443) from anywhere
  Egress:
    - DynamoDB VPC endpoint
    - Elasticsearch cluster

API Gateway Security Group:
  Ingress:
    - HTTPS (443) from anywhere
  Egress: None (managed service)

Elasticsearch Security Group:
  Ingress:
    - Port 443 from Lambda SG
    - Port 9200 from Lambda SG
  Egress: None

WAF Protection:

WAF Rules:
  - Name: "AWSManagedRulesCommonRuleSet"
    Priority: 1
    Statement:
      ManagedRuleGroupStatement:
        VendorName: "AWS"
        Name: "AWSManagedRulesCommonRuleSet"

  - Name: "RateLimitRule"
    Priority: 2
    Statement:
      RateBasedStatement:
        Limit: 2000
        AggregateKeyType: "IP"

When dealing with highly complex environments and high data volumes, the response time between identifying an anomaly and taking action is crucial. More than just a functional architecture, what is presented here is an approach focused on operational reliability, processing scalability, and cost control.

Each component has been integrated with a purpose: to accurately detect deviations, trigger automated responses, and maintain active governance at all levels from security to resource allocation. This type of solution becomes truly effective when aligned with the organization's technical routine and business objectives. It is not just about technology, but about delivering predictability in scenarios where failures are costly.

Thank you, see you next time.

AWS Copilot Deployment Tutorial: From Monolith to Microservice

Carlos Filho — Fri, 20 Jun 2025 18:09:54 +0000

This tutorial (for Mac) will guide you through deploying a Node.js application using AWS Copilot, first as a monolith and then breaking it down into microservices. By following these steps, you'll learn how to use AWS Copilot to deploy containerized applications to Amazon ECS.
I will leave a doc with all systems contemplated. Link: All Systems (Linux & Windows)

Tutorial Modules Overview

Introduction: Understanding the application and architecture
Setup: Installing and configuring prerequisites
Containerize and Deploy the Monolith: Building and deploying the application as a monolith
Break the Monolith: Separating the application into microservices
Deploy Microservices: Deploying each microservice independently
Clean Up: Removing all resources to avoid ongoing charges

Prerequisites

Before starting this tutorial, ensure you have the following:

AWS Account: You need an active AWS account. Some services may require your account to be active for more than 12 hours after creation.
Docker: Docker must be installed and running on your system.

For macOS:

Install Docker Desktop for Mac

Download from https://www.docker.com/products/docker-desktop/

Verify Docker installation

docker --version

Install AWS CLI v2 (for macOS)

curl "https://awscli.amazonaws.com/AWSCLIV2.pkg" -o "AWSCLIV2.pkg"
sudo installer -pkg AWSCLIV2.pkg -target /

Verify installation

aws --version

Configure AWS credentials

aws configure

AWS Copilot CLI: Install the AWS Copilot CLI

Install via Homebrew (macOS)

brew install aws/tap/copilot-cli

Verify installation

copilot --version

Part 1: Deploying the Monolithic Application

Step 1: Create an AWS Copilot Application

An AWS Copilot Application is a collection of services and environments.

# Clone the sample repository
git clone https://github.com/awslabs/amazon-ecs-nodejs-microservices
cd ./amazon-ecs-nodejs-microservices/

# Initialize a new Copilot application
copilot app init
# Name the application "api" when prompted

Step 2: Create the Environment

Copilot environments represent the infrastructure where your application runs.

# Create a new environment
copilot env init

When prompted:
Name the environment "api"
Choose "profile default" for credentials
Select "Yes, use default" for environment configuration

Step 3: Deploy the Environment

# Deploy the environment
copilot env deploy --name api

This step creates the VPC, subnets, security groups, and other necessary infrastructure for your application.

Step 4: Create the Monolith AWS Copilot Service

A Copilot Service runs containers on ECS or Fargate.

# Create a new service
copilot svc init

When prompted:
Select "Load Balanced Web Service" as the workload type
Name the service "monolith"
For the Dockerfile path, select or enter the appropriate path to the Dockerfile

Step 5: Deploy the Monolith Service

# Deploy the API
copilot env deploy --name api

Deploy the service

copilot svc deploy --name monolith

This builds the Docker container locally, pushes it to an Amazon Elastic Container Registry (ECR) repository, and deploys it on Amazon Elastic Container Service (ECS).

Step 6: Confirm the Deployment

After deployment completes, AWS Copilot will display a URL for your service. You can test the deployment by accessing the URL in a browser or using curl.
You can test the application deployment by entering queries in a browser, such as:

http:///api-ap-Publi-tqekqzwAKyhA-2052337664.us-east-1.elb.amazonaws.com/api/users/3
http:///api-ap-Publi-tqekqzwAKyhA-2052337664.us-east-1.elb.amazonaws.com/api/threads/2
http:///api-ap-Publi-tqekqzwAKyhA-2052337664.us-east-1.elb.amazonaws.com/api/posts/
http:///api-ap-Publi-tqekqzwAKyhA-2052337664.us-east-1.elb.amazonaws.com/api/posts/in-thread/1

Part 2: Breaking the Monolith into Microservices

Step 1: Create the Microservices

Create separate services for posts, threads, and users components.

Create the posts service (copy all the script up to Load Balanced Web Service)

copilot svc init --app api --dockerfile ./3-microservices/services/posts/Dockerfile --name posts --svc-type "Load Balanced Web Service"

Create the threads service (copy all the script up to Load Balanced Web Service)

copilot svc init --app api --dockerfile ./3-microservices/services/threads/Dockerfile --name threads --svc-type "Load Balanced Web Service"

Create the users service (copy all the script up to Load Balanced Web Service)

copilot svc init --app api --dockerfile ./3-microservices/services/users/Dockerfile --name users --svc-type "Load Balanced Web Service"

Step 2: Edit the Path in the Manifest Files

AWS Copilot sets the path to a service based on the service name. Edit each microservice manifest to ensure proper routing.

Edit the posts service manifest

nano copilot/posts/manifest.yml

Update the path under the http section to:

path: "api/posts"

Repeat for the threads and users services, updating their respective paths to:

For threads service

nano copilot/threads/manifest.yml

Edit to:

path: "api/threads"

For users service

nano copilot/users/manifest.yml

Edit to:

path: "api/users"

Step 3: Deploy the Microservices

Deploy the posts service

copilot svc deploy --name posts

Your service is accessible at http://api-ap-Publi-tqekqzwAKyhA-2052337664.us-east-1.elb.amazonaws.com/api/posts over the internet.

Deploy the threads service

copilot svc deploy --name threads

Your service is accessible at http://api-ap-Publi-tqekqzwAKyhA-2052337664.us-east-1.elb.amazonaws.com/api/threads over the internet.

Deploy the users service

copilot svc deploy --name users

Your service is accessible at http://api-ap-Publi-tqekqzwAKyhA-2052337664.us-east-1.elb.amazonaws.com/api/users over the internet.

Step 4: Shut Down the Monolith

Once all microservices are deployed and working, you can shut down the monolith service.

Delete the monolith service

copilot svc delete --name monolith

Step 5: Verify the Deployment

Test the deployment by accessing the endpoints:

Users endpoint: http://api-ap-Publi-tqekqzwAKyhA-2052337664.us-east-1.elb.amazonaws.com/api/users/3

Threads endpoint: http://api-ap-Publi-tqekqzwAKyhA-2052337664.us-east-1.elb.amazonaws.com/api/threads/2

Posts endpoints:http://api-ap-Publi-tqekqzwAKyhA-2052337664.us-east-1.elb.amazonaws.com/api/posts/

http://api-ap-Publi-tqekqzwAKyhA-2052337664.us-east-1.elb.amazonaws.com/api/posts/in-thread/1

Environment on AWS:

VPC:

ECS:

ELB:

Step 6: Clean Up Resources

Best practice is to delete resources when the tutorial is complete to avoid ongoing charges for running services. This cleanup process will terminate all the resources you created during this tutorial.

Step 1: Delete the Application

To delete the entire application and all associated resources:

Delete all resources

copilot app delete --name api

When you run this command:

AWS Copilot will list all the resources that will be deleted
You will be asked to confirm the deletion

The command will delete:

All services running on Amazon ECS
The Application Load Balancer
Amazon Elastic Container Registry repositories
AWS CloudFormation stacks
Underlying EC2 instances This process takes approximately 10 minutes to complete.

Resources Being Deleted
This cleanup will remove:

Amazon Elastic Container Service resources
Amazon Elastic Container Registry repositories
Elastic Load Balancer
Associated CloudFormation stacks
IAM roles created for your services
CloudWatch logs
Any other resources created by AWS Copilot

Thank you a lot! See you soon. I hope you enjoy it!

AWS Resource Explorer with PrivateLink

Carlos Filho — Tue, 10 Jun 2025 19:58:28 +0000

AWS continues to expand its offer of resource visibility with enhanced security. The newest addition to the arsenal is the integration between AWS Resource Explorer and AWS PrivateLink, now available in all commercial regions. This move marks an important step for organizations that need to locate resources broadly, but without giving up private traffic, especially in environments with tighter security restrictions.

Resource Explorer alone represents a breakthrough in the way administrators view their resources spread across different accounts and regions. Now, with support for PrivateLink, this capability can be achieved within the scope of VPC, without the need to expose calls to the public Internet.

This type of evolution addresses a sensitive point in multi-account architectures: how to ensure that distributed teams have visibility over the environment without compromising private connectivity standards? The new integration solves precisely this dilemma. It enables broad searches for resources using the Console, CLI, or SDKs, all encapsulated within the private VPC fabric.

In practice, this means that it is possible to consult resources on an organizational scale, with enhanced security and without relying on public routes. The experience is still accessible via the unified search bar, which maintains operational fluidity even in environments with tighter governance.

The big advantage here is not just in the discovery itself, but in the way this discovery is made. By keeping traffic within the PrivateLink domain, the risk of exposing sensitive metadata and API calls is drastically reduced. This control is essential for companies that need to meet more demanding compliance requirements, especially in sectors such as finance, healthcare, or government.

With this update, Resource Explorer takes another leap towards becoming not just an inventory tool, but a central point of contextual visibility, and now more compatible with corporate security policies.

I hope you enjoyed it! See you next time.

DEV Community: Carlos Filho

Scheduling weekly SNS notifications with Amazon EventBridge

What had to change, and what could not

Requirements for the weekly trigger

Four designs for the same weekly trigger

Cron on a dedicated EC2 instance

A fanout subscription on the existing topic

A Step Functions state machine on a schedule

An EventBridge rule publishing straight to the topic

Choosing the smallest operational surface

The publish path, from schedule to subscriber

EventBridge cron expressions run in UTC

The CloudFormation template for the EventBridge rule

Parameter validation that fails the deploy instead of the campaign

Whether the stack owns the campaign topic

Encryption at rest, and why the AWS managed key does not work here

The topic policy, and what the origin conditions prevent

The input transformer and the execution identifier

Explicit retry policy and dead letter queue

Four alarms and the TreatMissingData detail

Deploying the stack

Proving the schedule works without waiting for Saturday

The EventBridge Scheduler version, with a native time zone

Trade-offs, quotas and production concerns

Delivering to millions of subscribers

How long SNS keeps retrying a delivery, by endpoint type

At-least-once delivery and duplicate campaigns

Metrics, alarms and what is opt-in

What dominates the bill

Least privilege, encryption and the account boundary

Where this design stops serving

What the client ended up with

References

Blocking Container deployments based on vulnerability scans

Prerequisites

The problem

The solutions

ECR Enhanced Scanning with Inspector + EventBridge gate

Inspector findings to Security Hub with cross-account aggregation

Full variable list

Outputs

Validating the deployment

Why the other options do not work?

Architecture

Cost to keep in mind

RDS High Availability and credential rotation without downtime

Prerequisites

The problem

The solutions

Multi-AZ + RDS Proxy

Secrets Manager with automatic rotation

Terraform

File structure

variables.tf

networking.tf: security groups

iam.tf: roles for RDS Proxy and rotation Lambda

rds.tf: RDS instance and Proxy

secrets.tf: Secrets Manager and rotation

outputs.tf

Validating the deployment

Why the other options do not work?

What the architecture looks like end to end

Cost to keep in mind

Enforcing SSE-KMS on S3 buckets across all AWS accounts

Prerequisites

How does the solution work?

SCP: the preventive layer

AWS Config: the detective and corrective layer

Terraform

File structure

variables.tf

scp.tf: create and attach the SCP

iam.tf: roles for Config and SSM remediation

config.tf - Config rule and remediation for the management account

stackset.tf: deploy Config rule to all member accounts

What happens after deployment?

Validating the deployment

A note on the KMS key

Cost considerations

Why not the other options?

`variables.tf`

`networking.tf`: security groups

`iam.tf`: roles for RDS Proxy and rotation Lambda

`rds.tf`: RDS instance and Proxy

`secrets.tf`: Secrets Manager and rotation

`outputs.tf`

`variables.tf`

`scp.tf`: create and attach the SCP

`iam.tf`: roles for Config and SSM remediation

`config.tf` - Config rule and remediation for the management account

`stackset.tf`: deploy Config rule to all member accounts