DEV Community: Todor Todorov

Proactive RDS Monitoring: How We Built an EventBridge-Powered Alert System for Flawless BFCM Events

Todor Todorov — Fri, 07 Nov 2025 13:02:31 +0000

Stop database issues before they become customer-facing problems

The Problem: Silent Database Failures During Peak Traffic

Picture this: It's Black Friday, and your e-commerce platform is handling 10x the normal traffic. Everything seems fine—your application servers are scaling beautifully, your CDN is caching like a champ, and your monitoring dashboards show green across the board.

Then, suddenly, customer complaints start flooding in. Orders are timing out. Checkout pages are loading slowly. Your database is silently struggling, but you don't know it yet.

The culprit? RDS performance degradation that wasn't being caught by traditional monitoring.

Why Traditional Monitoring Falls Short

AWS RDS generates events for critical issues, but these events often go unnoticed:

RDS-EVENT-0189: Burst balance exhaustion (IOPS performance degradation)
RDS-EVENT-0225: Storage threshold at 80% (approaching auto-scaling limits)

These events are published to EventBridge, but without proper routing, they're just... sitting there. By the time you notice performance degradation in your application metrics, it's already impacting users.

During high-traffic events like Black Friday Cyber Monday (BFCM), this delay can mean the difference between a successful sale and a lost customer.

The Solution: EventBridge + Lambda + Slack = Proactive Alerts

We built a lightweight, serverless monitoring system that:

✅ Catches issues immediately - EventBridge triggers Lambda within seconds of RDS events

✅ Provides actionable context - Rich Slack notifications with step-by-step resolution guides

✅ Scales automatically - No infrastructure to manage, pay-per-use pricing

✅ Integrates seamlessly - Works with existing Terraform infrastructure

Architecture Overview

┌─────────────┐
│   AWS RDS   │ ──[Event]──> ┌──────────────┐
│  Instance   │              │ EventBridge  │
└─────────────┘              └──────┬───────┘
                                    │
                                    │ Triggers
                                    ▼
                            ┌─────────────────┐
                            │  Lambda Function│
                            │  (Python 3.12)  │
                            └───────┬─────────┘
                                    │
                                    │ Formats & Sends
                                    ▼
                            ┌─────────────────┐
                            │  Slack Channel  │
                            │  (Rich Alerts)  │
                            └─────────────────┘

Key Components:

EventBridge Rule: Filters for RDS-EVENT-0189 and RDS-EVENT-0225
Lambda Function: Processes events and formats Slack messages
SSM Parameter Store: Securely stores Slack webhook URL
CloudWatch Logs: Captures all Lambda execution logs

Implementation: Infrastructure as Code

Let's build this step by step. We'll use Terraform for infrastructure and Python for the Lambda function.

Step 1: Create the Terraform Module

First, let's create a reusable Terraform module:

# terraform/modules/rds-important-events-monitor/main.tf

resource "aws_iam_role" "rds_important_events_monitor" {
  count = var.create_iam_role ? 1 : 0

  name = "${var.function_name}-role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = "sts:AssumeRole"
        Effect = "Allow"
        Principal = {
          Service = "lambda.amazonaws.com"
        }
      }
    ]
  })
}

resource "aws_iam_role_policy" "lambda_logs" {
  count = var.create_iam_role ? 1 : 0

  name = "${var.function_name}-logs-policy"
  role = aws_iam_role.rds_important_events_monitor[0].id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Action = [
          "logs:CreateLogGroup",
          "logs:CreateLogStream",
          "logs:PutLogEvents"
        ]
        Resource = "arn:aws:logs:*:${data.aws_caller_identity.current.account_id}:*"
      }
    ]
  })
}

resource "aws_iam_role_policy" "ssm_read" {
  count = var.create_iam_role ? 1 : 0

  name = "${var.function_name}-ssm-policy"
  role = aws_iam_role.rds_important_events_monitor[0].id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Action = [
          "ssm:GetParameter",
          "ssm:GetParameters"
        ]
        Resource = "arn:aws:ssm:${var.aws_region}:${data.aws_caller_identity.current.account_id}:parameter${var.slack_webhook_url_parameter}"
      }
    ]
  })
}

resource "aws_lambda_function" "rds_important_events_monitor" {
  depends_on = [
    aws_cloudwatch_log_group.lambda_logs
  ]

  function_name = var.function_name
  description   = "Monitor RDS issues and send Slack notifications"
  role          = var.create_iam_role ? aws_iam_role.rds_important_events_monitor[0].arn : var.lambda_role_arn
  handler       = var.handler
  runtime       = var.runtime
  timeout       = var.timeout
  memory_size   = var.memory_size

  filename         = "${var.source_path}/../lambda_function.zip"
  source_code_hash = filebase64sha256("${var.source_path}/../lambda_function.zip")

  environment {
    variables = {
      SLACK_WEBHOOK_URL = data.aws_ssm_parameter.slack_webhook_url.value
    }
  }

  tags = {
    Name        = var.function_name
    Description = "Monitors RDS events and sends Slack notifications"
  }
}

resource "aws_cloudwatch_log_group" "lambda_logs" {
  name              = "/aws/lambda/${var.function_name}"
  retention_in_days = var.cloudwatch_logs_retention_in_days
}

# EventBridge Rule to capture RDS events
resource "aws_cloudwatch_event_rule" "rds_iops_events" {
  name        = "${var.function_name}-events"
  description = "Capture RDS events RDS-EVENT-0189 and RDS-EVENT-0225 for IOPS-related/StorageSize issues"

  event_pattern = jsonencode({
    source      = ["aws.rds"]
    detail-type = ["RDS DB Instance Event"]
    detail = {
      EventID = [
        "RDS-EVENT-0189",
        "RDS-EVENT-0225"
      ]
    }
  })

  tags = {
    Name = "${var.function_name}-events"
  }
}

# Connect EventBridge to Lambda
resource "aws_cloudwatch_event_target" "lambda" {
  target_id = "RDSIOPSMonitorLambdaTarget"
  rule      = aws_cloudwatch_event_rule.rds_iops_events.name
  arn       = aws_lambda_function.rds_important_events_monitor.arn
}

# Allow EventBridge to invoke Lambda
resource "aws_lambda_permission" "allow_eventbridge" {
  statement_id  = "AllowExecutionFromEventBridge"
  action        = "lambda:InvokeFunction"
  function_name = aws_lambda_function.rds_important_events_monitor.function_name
  principal     = "events.amazonaws.com"
  source_arn    = aws_cloudwatch_event_rule.rds_iops_events.arn
}

# Data source to fetch Slack webhook from SSM
data "aws_ssm_parameter" "slack_webhook_url" {
  name = var.slack_webhook_url_parameter
}

data "aws_caller_identity" "current" {}

Step 2: Module Variables

# terraform/modules/rds-important-events-monitor/vars.tf

variable "function_name" {
  description = "Name of the Lambda function"
  type        = string
  default     = "rds-important-events-monitor"
}

variable "runtime" {
  description = "Lambda runtime"
  type        = string
  default     = "python3.12"
}

variable "source_path" {
  description = "Path to the Lambda source code directory"
  type        = string
}

variable "handler" {
  description = "Lambda handler"
  type        = string
  default     = "app.lambda_handler"
}

variable "timeout" {
  description = "Lambda timeout in seconds"
  type        = number
  default     = 30
}

variable "memory_size" {
  description = "Lambda memory size in MB"
  type        = number
  default     = 256
}

variable "slack_webhook_url_parameter" {
  description = "SSM Parameter Store path for Slack webhook URL"
  type        = string
  default     = "/us-east-1/rds-monitor/slack-webhook-url"
}

variable "aws_region" {
  description = "AWS region"
  type        = string
  default     = "us-east-1"
}

variable "cloudwatch_logs_retention_in_days" {
  description = "CloudWatch Logs retention in days"
  type        = number
  default     = 7
}

variable "create_iam_role" {
  description = "Whether to create an IAM role for the Lambda function"
  type        = bool
  default     = true
}

variable "lambda_role_arn" {
  description = "ARN of an existing IAM role to use for the Lambda function"
  type        = string
  default     = null
}

Step 3: Use the Module

# terraform/us-east-1/rds-important-events-monitor.tf

module "rds_important_events_monitor" {
  source          = "../modules/rds-important-events-monitor"
  source_path     = "../../rds-important-events-monitor/src"
  create_iam_role = false
  lambda_role_arn = "arn:aws:iam::YOUR_ACCOUNT_ID:role/rds-important-events-monitor"
}

The Lambda Function: Intelligent Alert Formatting

The Lambda function does more than just forward events—it provides actionable, context-rich notifications that help your team respond quickly.

Complete Lambda Code

# rds-important-events-monitor/src/app.py

import json
import os
import logging
import requests

logger = logging.getLogger()
logger.setLevel(logging.INFO)

aws_region = os.environ.get('AWS_REGION', 'us-east-1')


def lambda_handler(event, context):
    logger.info(f"Received event: {json.dumps(event)}")

    try:
        detail = event.get('detail', {})
        source_identifier = detail.get('SourceIdentifier', '')
        event_message = detail.get('Message', '')
        event_id = detail.get('EventID', '')
        timestamp = event.get('time', '')
        region = event.get('region', aws_region)
        account_id = event.get('account', '')

        expected_events = ['RDS-EVENT-0189', 'RDS-EVENT-0225']
        if event_id not in expected_events:
            logger.warning(f"Received unexpected event ID: {event_id}, expected one of {expected_events}")
            return {
                'statusCode': 200,
                'body': json.dumps(f'Unexpected event ID: {event_id}')
            }

        db_name = source_identifier

        if not db_name:
            logger.warning("Could not extract DB name from event")
            return {
                'statusCode': 200,
                'body': json.dumps('Event processed but no DB name found')
            }

        logger.info(f"Processing {event_id} for database: {db_name}")

        message = format_rds_event_slack_message(
            db_name=db_name,
            event_message=event_message,
            event_id=event_id,
            timestamp=timestamp,
            region=region,
            account_id=account_id
        )
        success = post_to_slack(message)

        if success:
            logger.info(f"Successfully sent Slack notification for {db_name}")
        else:
            logger.error(f"Failed to send Slack notification for {db_name}")

        return {
            'statusCode': 200 if success else 500,
            'body': json.dumps({
                'message': 'Notification sent' if success else 'Notification failed',
                'db_name': db_name,
                'event_message': event_message
            })
        }

    except Exception as e:
        logger.error(f"Error processing event: {str(e)}", exc_info=True)
        return {
            'statusCode': 500,
            'body': json.dumps(f'Error: {str(e)}')
        }


def format_rds_event_slack_message(db_name, event_message, event_id, timestamp, region, account_id):
    """Format RDS event into a rich Slack message with actionable guidance"""

    if event_id == 'RDS-EVENT-0189':
        title = f"🚨 RDS Burst Balance Alert (0189): {db_name}"
        description = (
            "*📋 What This Means:*\n"
            "The RDS instance has exhausted its burst balance for General Purpose (SSD) storage. "
            "Burst balance is a performance credit system that allows temporary spikes in I/O performance. "
            "When depleted, the instance will be limited to baseline IOPS, which can cause significant "
            "performance degradation and application slowdowns.\n\n"

            "*⚠️ Impact:*\n"
            "• Database queries may become significantly slower\n"
            "• Application response times will increase\n"
            "• User experience will be negatively affected\n"
            "• If not addressed, the database may become unresponsive during peak loads\n\n"

            "*🔧 Resolution Steps:*\n"
            "1. *Immediate Action:* Navigate to the RDS Console and review the instance's current storage configuration\n"
            "2. *Check Current Setup:*\n"
            "   • Review current storage type (likely gp2/gp3)\n"
            "   • Check current IOPS configuration\n"
            "   • Review CloudWatch metrics for I/O patterns\n"
            "3. *Choose Resolution Path:*\n"
            "   • *Option A - Upgrade to Provisioned IOPS (io1/io2):* Best for consistent high-performance workloads\n"
            "     - Go to Modify → Storage → Change storage type to io1 or io2\n"
            "     - Set appropriate Provisioned IOPS (typically 3x-5x your current baseline)\n"
            "     - Apply changes during maintenance window or immediately if critical\n"
            "   • *Option B - Upgrade to gp3 with higher baseline:* Good balance of cost and performance\n"
            "     - Modify storage type to gp3\n"
            "     - Increase baseline IOPS (default 3000, can go up to 16000)\n"
            "     - Increase throughput if needed (up to 1000 MiB/s)\n"
            "4. *Monitor:* After changes, monitor CloudWatch metrics to ensure performance improves\n"
        )
    elif event_id == 'RDS-EVENT-0225':
        title = f"🚨 RDS Storage Threshold Alert (0225): {db_name}"
        description = (
            "*📋 What This Means:*\n"
            "The RDS instance storage has reached 80% of its maximum allocated storage capacity. "
            "While RDS supports storage auto-scaling, this threshold indicates that auto-scaling is "
            "approaching its configured maximum limit. If storage continues to grow, you may hit the "
            "maximum storage limit, which could cause the database to become read-only or unavailable.\n\n"

            "*⚠️ Impact:*\n"
            "• Database may become read-only if storage reaches 100%\n"
            "• New writes will fail, causing application errors\n"
            "• Auto-scaling may not be able to expand further if max storage is reached\n"
            "• Potential data loss risk if not addressed promptly\n\n"

            "*🔧 Resolution Steps:*\n"
            "1. *Immediate Assessment:*\n"
            "   • Check current storage usage in RDS Console\n"
            "   • Review storage auto-scaling configuration\n"
            "   • Identify what's consuming storage space\n"
            "2. *Investigate Storage Usage:*\n"
            "   • Connect to the database and check table sizes\n"
            "   • Look for large tables, indexes, or temporary files\n"
            "   • Check for unoptimized queries creating large temp tables\n"
            "   • Review binary logs, error logs, and slow query logs size\n"
            "3. *Immediate Actions (if needed):*\n"
            "   • *Increase Max Storage:* Go to Modify → Storage → Increase Max Storage\n"
            "     - Increase by at least 20-30% to provide buffer\n"
            "     - Consider future growth projections\n"
            "   • *Clean Up Space (if applicable):*\n"
            "     - Archive or delete old data\n"
            "     - Optimize tables (ANALYZE/OPTIMIZE)\n"
            "     - Remove unnecessary indexes\n"
            "     - Clean up old binary logs (if safe to do so)\n"
            "4. *Review Auto-Scaling Settings:*\n"
            "   • Ensure auto-scaling is enabled\n"
            "   • Verify max storage limit is appropriate for your needs\n"
            "   • Consider increasing max storage proactively\n"
        )
    else:
        # Fallback for any other events
        title = f"🚨 RDS Event Alert ({event_id}): {db_name}"
        description = (
            f"*Event Details:*\n"
            f"{event_message}\n\n"
            "*⚠️ Action Required:* Please review this event and take appropriate action."
        )

    text_parts = [
        f"*Event ID:* `{event_id}`",
        f"*Account ID:* `{account_id}`",
        f"*RDS Name:* `{db_name}`",
        f"*Region:* `{region}`",
        f"*Issue:* {event_message}",
        "",
        description
    ]

    if timestamp:
        text_parts.insert(-1, f"*Time:* {timestamp}")

    rds_console_link = f"https://{region}.console.aws.amazon.com/rds/home?region={region}#database:id={db_name};is-cluster=false"
    cloudwatch_link = f"https://{region}.console.aws.amazon.com/cloudwatch/home?region={region}#metricsV2:graph=~();query=AWS/RDS%20{db_name}"

    return {
        'attachments': [{
            'fallback': title,
            'color': 'danger',
            'title': title,
            'text': '\n'.join(text_parts),
            'fields': [
                {
                    'title': 'Quick Actions',
                    'value': f"<{rds_console_link}|View in RDS Console> | <{cloudwatch_link}|View CloudWatch Metrics>",
                    'short': False
                }
            ],
            'footer': 'RDS Important Events Monitor'
        }]
    }


def post_to_slack(message):
    """Send message to Slack via webhook"""
    webhook_url = os.environ.get('SLACK_WEBHOOK_URL')
    if not webhook_url:
        logger.error("SLACK_WEBHOOK_URL environment variable not set")
        return False

    try:
        response = requests.post(
            webhook_url,
            data=json.dumps(message),
            headers={'Content-Type': 'application/json'},
            timeout=10
        )

        if response.status_code == 200:
            logger.info("Successfully posted to Slack")
            return True
        else:
            logger.error(f"Failed to post to Slack: HTTP {response.status_code} - {response.text}")
            return False

    except Exception as e:
        logger.error(f"Exception posting to Slack: {str(e)}")
        return False

Dependencies

# rds-important-events-monitor/src/requirements.txt

requests==2.31.0

Setting Up Slack Integration

Create a Slack Incoming Webhook:
- Go to your Slack workspace settings
- Navigate to Apps → Incoming Webhooks
- Create a new webhook for your monitoring channel
- Copy the webhook URL
Store the Webhook in SSM Parameter Store:

   aws ssm put-parameter \
     --name "/us-east-1/rds-monitor/slack-webhook-url" \
     --value "https://hooks.slack.com/services/YOUR/WEBHOOK/URL" \
     --type "SecureString" \
     --region us-east-1

Deploy with Terraform:

   cd terraform/us-east-1
   terraform init
   terraform plan
   terraform apply

What the Slack Alert Looks Like

When an RDS event occurs, your team receives a rich Slack message like this:

🚨 RDS Burst Balance Alert (0189): production-db-instance

Event ID: RDS-EVENT-0189
Account ID: 123456789012
RDS Name: production-db-instance
Region: us-east-1
Issue: Your database instance has exhausted its burst balance...

📋 What This Means:
The RDS instance has exhausted its burst balance for General Purpose (SSD) storage...

⚠️ Impact:
• Database queries may become significantly slower
• Application response times will increase
...

🔧 Resolution Steps:
1. Immediate Action: Navigate to the RDS Console...
2. Check Current Setup: ...
3. Choose Resolution Path: ...

Quick Actions: [View in RDS Console] | [View CloudWatch Metrics]

Key Features:

🎨 Color-coded (red for danger)
📋 Context about what the event means
⚠️ Clear impact assessment
🔧 Step-by-step resolution guide
🔗 Direct links to AWS Console and CloudWatch

Benefits: Why This Matters for BFCM

1. Proactive Problem Detection

Instead of waiting for customer complaints or application metrics to degrade, you're notified immediately when RDS issues occur. During BFCM, this can save hours of troubleshooting.

2. Actionable Alerts

The alerts don't just say "something's wrong"—they provide:

Context about what's happening
Impact assessment
Step-by-step resolution instructions
Direct links to relevant AWS resources

3. Team Collaboration

Slack notifications ensure the right people see the issue immediately. No more checking CloudWatch dashboards or waiting for automated reports.

4. Cost-Effective

Lambda: Pay only for invocations (typically $0.20 per million requests)
EventBridge: First 5 million events/month are free
Total monthly cost: Usually under $1 for most workloads

5. Scalable Architecture

The system automatically handles:

Multiple RDS instances
High event volumes during peak traffic
Regional deployments

6. Infrastructure as Code

Everything is version-controlled and reproducible:

Easy to deploy to new environments
Simple to modify and extend
No manual configuration drift

Real-World Impact

Before: During a previous BFCM event, we discovered RDS performance issues 2 hours after they started, leading to:

3-hour incident response time

After: With this monitoring system:

Issues detected within 30 seconds
Team notified immediately via Slack
Resolution time reduced to 15 minutes
Zero customer-facing impact

Extending the Solution

This pattern can be extended to monitor other critical AWS events:

# Example: Add ECS service deployment failures
resource "aws_cloudwatch_event_rule" "ecs_deployment_failures" {
  name = "ecs-deployment-failures"

  event_pattern = jsonencode({
    source      = ["aws.ecs"]
    detail-type = ["ECS Deployment State Change"]
    detail = {
      eventName = ["SERVICE_DEPLOYMENT_FAILED"]
    }
  })
}

Or add more RDS events:

detail = {
  EventID = [
    "RDS-EVENT-0189",  # Burst balance
    "RDS-EVENT-0225",  # Storage threshold
    "RDS-EVENT-0169",  # DB instance restart
    "RDS-EVENT-0151"   # DB instance availability
  ]
}

Best Practices

Test Your Alerts: Use AWS CLI to simulate events:

   aws events put-events --entries '[
     {
       "Source": "aws.rds",
       "DetailType": "RDS DB Instance Event",
       "Detail": "{\"EventID\":\"RDS-EVENT-0189\",\"SourceIdentifier\":\"test-db\",\"Message\":\"Test event\"}"
     }
   ]'

Monitor the Monitor: Set up CloudWatch alarms for Lambda errors
Rotate Secrets: Regularly rotate your Slack webhook URL
Review Logs: Periodically review CloudWatch logs for patterns
Update Documentation: Keep resolution steps current as your infrastructure evolves

Conclusion

Proactive monitoring isn't just about catching problems—it's about preventing them from impacting your customers. During critical events like BFCM, every second counts.

This EventBridge + Lambda + Slack solution gives you:

✅ Immediate notification of RDS issues
✅ Actionable, context-rich alerts
✅ Zero infrastructure management
✅ Cost-effective scaling
✅ Infrastructure as Code best practices

The result? A monitoring system that works silently in the background, alerting your team only when action is needed, with all the context they need to resolve issues quickly.

Resources

Have you built similar monitoring solutions? Share your experiences in the comments below! 🚀

How to Migrate ECR Docker Images Between Repositories (with Automation)

Todor Todorov — Wed, 24 Sep 2025 08:49:18 +0000

Every now and then, developers find themselves in a situation where they need to migrate container images between Amazon ECR repositories.

Maybe the naming convention changed.
Maybe a new repo structure was introduced.
Maybe you just want to clean things up.

Whatever the reason, doing this manually — one image at a time — is slow and painful. And in fast-moving teams, anything that slows us down affects our velocity and robustness.

That’s why I built an automation script to migrate images quickly and safely. It uses skopeo under the hood to copy images between repositories without pulling and pushing them manually.

Let’s dive in. 👇

🛠️ Prerequisites

AWS CLI configured with a profile that has access to the ECR repositories
jq for JSON parsing
skopeo for copying container images

Installing `skopeo`

macOS (Homebrew)

  brew install skopeo

Ubuntu/Debian

  sudo apt-get update
  sudo apt-get -y install skopeo

Windows (via WSL2 or Chocolatey)
- If you’re using WSL2 (Ubuntu), just use the Ubuntu instructions.
- If you’re on Windows with Chocolatey:
```
choco install skopeo
```

📜 The Migration Script

Below is the script I use on macOS (works with GNU/Linux and WSL as well, only date syntax differs).

It migrates all images pushed within the last N days from a source ECR repo to a destination ECR repo.

👉 Save it as migrate-ecr-images.sh and run it.

#!/usr/bin/env bash
set -euo pipefail

# --- check for skopeo and install if missing (macOS Homebrew) ---
if ! command -v skopeo >/dev/null 2>&1; then
  echo "skopeo not found, installing..."
  if ! command -v brew >/dev/null 2>&1; then
    echo "Homebrew is required but not found. Please install Homebrew first: https://brew.sh/"
    exit 1
  fi
  brew install skopeo
fi

SRC_REPO="java-micro-repo"
DST_REPO="repo"
WINDOW_DAYS=90
DRY_RUN=false

# --- detect env from AWS profile ---
REGION="${AWS_REGION:-${AWS_DEFAULT_REGION:-$(aws configure get region)}}"
: "${REGION:?Region not set in environment or AWS config}"
ACCOUNT_ID="$(aws sts get-caller-identity --query Account --output text)"
REG_HOST="${ACCOUNT_ID}.dkr.ecr.${REGION}.amazonaws.com"

# --- BSD/GNU date helpers ---
is_bsd_date=false
if date -v-1d +%s >/dev/null 2>&1; then is_bsd_date=true; fi
cutoff_epoch() {
  if $is_bsd_date; then date -u -v-"${WINDOW_DAYS}"d +%s; else date -u -d "${WINDOW_DAYS} days ago" +%s; fi
}
to_epoch() {
  ts="$1"
  if $is_bsd_date; then
    ts_norm=$(printf "%s" "$ts" \
      | sed -E 's/Z$/+0000/' \
      | sed -E 's/([+-][0-9]{2}):([0-9]{2})$/\1\2/' \
      | sed -E 's/([0-9]{2}:[0-9]{2}:[0-9]{2})\.[0-9]+/\1/')
    case "$ts_norm" in *+????|*-[0-9][0-9][0-9][0-9]) : ;; *) ts_norm="${ts_norm}+0000" ;; esac
    date -j -u -f "%Y-%m-%dT%H:%M:%S%z" "$ts_norm" +%s
  else
    date -u -d "$ts" +%s
  fi
}

CUTOFF_EPOCH="$(cutoff_epoch)"

echo "Source:      ${REG_HOST}/${SRC_REPO}"
echo "Destination: ${REG_HOST}/${DST_REPO}"
echo "Window:      last ${WINDOW_DAYS} days"
echo "Dry run:     ${DRY_RUN}"
echo

# --- ensure destination repo exists ---
if ! aws ecr describe-repositories --repository-names "$DST_REPO" >/dev/null 2>&1; then
  echo "Creating destination repo: $DST_REPO"
  aws ecr create-repository --repository-name "$DST_REPO" >/dev/null
  aws ecr put-image-tag-mutability --repository-name "$DST_REPO" --image-tag-mutability MUTABLE >/dev/null || true
fi

# --- get ECR password for skopeo auth ---
ECR_PASS="$(aws ecr get-login-password --region "$REGION")"

# --- collect recent tags ---
TMP_LIST="$(mktemp)"; TMP_TAGS="$(mktemp)"
aws ecr describe-images --repository-name "$SRC_REPO" --output json \
| jq -r '.imageDetails[] | select(.imageTags!=null) | .imagePushedAt as $t | .imageTags[] | "\($t)\t\(.)"' > "$TMP_LIST"

while IFS=$'\t' read -r pushed tag; do
  epoch="$(to_epoch "$pushed")" || continue
  if [ "$epoch" -ge "$CUTOFF_EPOCH" ]; then printf "%s\n" "$tag"; fi
done < "$TMP_LIST" | sort -u > "$TMP_TAGS"

if ! [ -s "$TMP_TAGS" ]; then
  echo "No images pushed in the last ${WINDOW_DAYS} days. Nothing to do."
  rm -f "$TMP_LIST" "$TMP_TAGS"; exit 0
fi

echo "Found $(wc -l < "$TMP_TAGS" | tr -d ' ') tag(s):"
sed 's/^/  - /' "$TMP_TAGS"
echo

FAILED=()
while IFS= read -r TAG; do
  [ -z "$TAG" ] && continue
  SRC="docker://${REG_HOST}/${SRC_REPO}:${TAG}"
  DST="docker://${REG_HOST}/${DST_REPO}:${TAG}"
  echo "Copying ${TAG} …"
  if [ "$DRY_RUN" = true ]; then
    echo "  [dry-run] skopeo copy --all ${SRC} -> ${DST}"
    continue
  fi
  if ! skopeo copy --all \
        --src-creds "AWS:${ECR_PASS}" \
        --dest-creds "AWS:${ECR_PASS}" \
        "$SRC" "$DST"; then
    echo "  ❌ Failed: ${TAG}"
    FAILED+=("$TAG")
  else
    echo "  ✅ Done: ${TAG}"
  fi
done < "$TMP_TAGS"

rm -f "$TMP_LIST" "$TMP_TAGS"

echo
if [ "${#FAILED[@]}" -gt 0 ]; then
  echo "Completed with failures:"
  printf '  - %s\n' "${FAILED[@]}"
  exit 1
else
  echo "All requested tags copied successfully."
fi

⚡ How It Works

Finds recent images
Using aws ecr describe-images, it filters tags by their imagePushedAt date within the last WINDOW_DAYS.
Creates the destination repo if missing
So you don’t have to do it manually.
Copies images with skopeo
No need to pull and re-push—skopeo copies directly between registries.
Supports dry-run mode
Set DRY_RUN=true to preview what will happen.

🔧 Example Usage

Copy the last 90 days of images from java-micro-repo to repo:

SRC_REPO="java-micro-repo" DST_REPO="repo" WINDOW_DAYS=90 ./migrate-ecr-images.sh

Run a dry run (no changes made):

DRY_RUN=true ./migrate-ecr-images.sh

✅ Why This Helps

Saves hours compared to manual copying
Automates repo creation if missing
Respects velocity → lets you focus on coding, not housekeeping
Cross-platform: works on macOS, Ubuntu, and Windows

🎯 Final Thoughts

This script became a huge time-saver in my team. Anytime we need to rename or restructure ECR repositories, it’s just one command away.

If you work with AWS ECR often, I recommend keeping this script handy in your toolbox. 🔧

👉 What about you — have you run into ECR image migration headaches?
Drop your thoughts in the comments—I’d love to hear how you’ve solved it.

(Near)Real-Time Subnet IP Monitoring with AWS Lambda and Slack Alerts

Todor Todorov — Fri, 03 Jan 2025 23:31:17 +0000

Subnet IP exhaustion can lead to operational bottlenecks and service outages in cloud environments. To address this challenge, we’ll build an AWS Lambda function to monitor the availability of free IP addresses in subnets across your VPCs and send real-time Slack alerts when the count drops below a threshold.

But wait a sec?! Doesn't AWS provide monitoring for this out of the box?!

Of course, they are providing VPC IPAM, and you may use it (to be honest I would prefer also to use it if there was no budget limitation). I always would prefer the managed solution, but this time, when we needed to get approval for a budget, if it was for 1000 IPs, it wasn't going to be a big deal because pricing is $0.00027 per hour per monitored IP by IPAM. However, in my case, with hundreds of thousands of IPs, the budget was not approved, and needed to think of something custom and cost-effective solution.

Key Features

Proactive Monitoring: Checks the available IPs in each subnet at regular intervals.
Real-Time Alerts: Sends Slack notifications when IP availability in a subnet is critically low.
Secure Configuration: Retrieves Slack webhook URL from AWS Secrets Manager, ensuring secure handling of sensitive data.

Prerequisites

AWS CLI installed and configured
AWS SAM CLI installed
Python 3.x
Basic understanding of AWS Lambda, Slack API, and VPC networking
A Slack webhook URL stored in AWS Secrets Manager

Step 1: Create the SAM Project

Initialize a new SAM project:

sam init

Choose the following options:

Template: AWS Quick Start Templates
Runtime: python3.9 (or the latest version)
Application: Hello World Example

Navigate to the project directory:

cd <your-project-directory>

Step 2: Implement the Lambda Code

Replace the contents of app.py with the following code:

import boto3
import logging
import requests
import os
from botocore.exceptions import ClientError

# Logging setup
logger = logging.getLogger()
logger.setLevel(logging.INFO)

# AWS client
ec2_client = boto3.client('ec2')

# Fetch the Slack webhook URL from environment variable
SLACK_WEBHOOK_URL = os.getenv('SLACK_WEBHOOK_URL')
if not SLACK_WEBHOOK_URL:
    raise ValueError("SLACK_WEBHOOK_URL environment variable is not set.")

def fetch_subnets():
    """Retrieve all subnets in all VPCs."""
    try:
        paginator = ec2_client.get_paginator('describe_subnets')
        for page in paginator.paginate():
            for subnet in page['Subnets']:
                yield subnet
    except ClientError as e:
        logger.error(f"Error fetching subnets: {e}")
        raise

def send_slack_notification(message):
    """Send a message to Slack."""
    try:
        response = requests.post(SLACK_WEBHOOK_URL, json={"text": message})
        if response.status_code != 200:
            logger.error(f"Failed to send Slack notification: {response.text}")
        else:
            logger.info("Slack notification sent successfully.")
    except Exception as e:
        logger.error(f"Error sending Slack notification: {e}")

def lambda_handler(event, context):
    """Main Lambda handler to monitor subnets and send alerts."""
    logger.info("Starting subnet monitoring.")

    try:
        for subnet in fetch_subnets():
            subnet_id = subnet['SubnetId']
            cidr_block = subnet['CidrBlock']
            available_ips = subnet['AvailableIpAddressCount']

            logger.info(f"Subnet {subnet_id} ({cidr_block}) has {available_ips} available IPs.")

            if available_ips < 10:
                message = (f"🚨 *Low IP Alert*: Subnet {subnet_id} ({cidr_block}) has only {available_ips} free IPs left. "
                           f"Immediate action may be required!")
                send_slack_notification(message)
    except Exception as e:
        logger.error(f"Unexpected error: {e}")

Step 3: Configure the SAM Template

Update template.yaml to define the Lambda function and required permissions:

Resources:
  SubnetMonitorFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: .
      Handler: app.lambda_handler
      Runtime: python3.12
      Timeout: 60
      Policies:
        - EC2ReadOnlyAccess
        - SecretsManagerReadWrite
      Environment:
        Variables:
          SLACK_WEBHOOK_URL: '{{resolve:secretsmanager:MySecret}}'
      Events:
        ScheduledEvent:
          Type: Schedule
          Properties:
            Schedule: rate(5 minutes)

Step 4: Build and Deploy

Build your application:

sam build

Deploy your application:

sam deploy --guided

During deployment, you’ll be prompted to provide a stack name and parameter values. Ensure that your Slack webhook URL is securely stored in Secrets Manager.

Step 5: Test and Monitor

After deployment:

Check the Lambda execution logs in CloudWatch Logs for subnet data and alert activity.
Trigger a test scenario by reducing the available IP count in a subnet (e.g., by launching instances).
Verify that alerts appear in your Slack channel.

Summary

By following this tutorial, you’ve created a serverless solution to:

Proactively monitor free IP availability in VPC subnets.
Notify your team on Slack when IP resources are critically low.

This system ensures you can address IP exhaustion issues before they impact your applications, giving you peace of mind in managing your cloud network resources.

Monitor EBS Volume Health

Todor Todorov — Fri, 03 Jan 2025 22:50:23 +0000

This blog post is inspired by an interesting question that was asked during the Ask The Experts panel at AWS Community Day Bulgaria 2024.
It’s a fascinating topic that I wanted to dive into and contribute about — a bit late, but now I have found the time to put it together, I hope you will find it useful.

In this tutorial, we’ll create and deploy an AWS Lambda function using the AWS Serverless Application Model (SAM) that:

Fetches health status metrics for EBS volumes.
Pushes the metrics to CloudWatch for monitoring and historical data.

This solution provides the ability to create alarms (if needed) and view historical data for specific EBS volumes (the main use case where the solution was searched for).

Prerequisites

AWS CLI installed and configured
AWS SAM CLI installed
Python 3.x
Basic knowledge of AWS Lambda and CloudWatch

Step 1: Create the Project

Start by creating a new SAM project:

sam init

Choose the following options:

Template: AWS Quick Start Templates
Runtime: python3.12 (or the latest version)
Application: Hello World Example
Project Name: ebs-monitor
For other prompts let's agree to use defaults

This generates a basic folder structure. Navigate to the project directory(the name of the project you set during init process):

cd <your-project-directory>

Step 2: Update the Lambda Code

Replace the contents of the app.py file with the following code:

import boto3
import logging
from botocore.exceptions import ClientError
from datetime import datetime

logger = logging.getLogger()
logger.setLevel(logging.INFO)

ec2_client = boto3.client('ec2')
cloudwatch_client = boto3.client('cloudwatch')

def fetch_ebs_volumes():
    """Fetch all EBS volumes, handling pagination."""
    paginator = ec2_client.get_paginator('describe_volumes')
    for page in paginator.paginate():
        for volume in page['Volumes']:
            yield volume

def publish_metrics(volume_id, status):
    """Publish the volume's health status to CloudWatch."""
    try:
        cloudwatch_client.put_metric_data(
            Namespace='EBS/Health',
            MetricData=[
                {
                    'MetricName': 'VolumeHealthStatus',
                    'Dimensions': [
                        {
                            'Name': 'VolumeId',
                            'Value': volume_id
                        }
                    ],
                    'Timestamp': datetime.utcnow(),
                    'Value': 1 if status == 'ok' else 0,
                    'Unit': 'Count'
                }
            ]
        )
        logger.info(f"Published health status for volume {volume_id}: {status}")
    except ClientError as e:
        logger.error(f"Failed to publish metrics for volume {volume_id}: {e}")

def lambda_handler(event, context):
    """Main Lambda function handler."""
    logger.info("Fetching EBS volume health status")
    volume_count = 0

    try:
        for volume in fetch_ebs_volumes():
            volume_id = volume['VolumeId']
            status = volume.get('State', 'unknown')
            publish_metrics(volume_id, status)
            volume_count += 1
        logger.info(f"Successfully processed health metrics for {volume_count} volumes.")
    except ClientError as e:
        logger.error(f"Error fetching EBS volumes: {e}")

Step 3: Update the Template

Edit the template.yaml file to define the Lambda function and required permissions:

Resources:
  EBSTrackerFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: .
      Handler: app.lambda_handler
      Runtime: python3.12
      Timeout: 60
      Policies:
        - EC2ReadOnlyAccess
        - CloudWatchPutMetricPolicy
      Events:
        ScheduledEvent:
          Type: Schedule
          Properties:
            Schedule: rate(5 minutes)

Step 4: Build and Deploy

Build the application:

sam build

Deploy the application:

sam deploy --guided

Follow the prompts to provide a stack name and configure deployment parameters.

Step 5: Verify and Monitor

Once deployed, the Lambda function will run every 5 minutes. Metrics for each EBS volume’s health status will be available in CloudWatch under the namespace EBS/Health.

Create Alarms

You can create alarms in CloudWatch based on the VolumeHealthStatus metric for specific EBS volumes to notify you of any issues.

Summary

In this tutorial, we:

Used AWS SAM to create and deploy a Lambda function.
Monitored EBS volumes.
Published health metrics to CloudWatch for historical tracking and/or monitoring.

This solution provides a scalable way to monitor your EBS volumes and maintain visibility into their health over time.

Automating Redrive from DLQ for FIFO SQS Queue

Todor Todorov — Thu, 13 Jul 2023 08:16:34 +0000

The case:

Dead Letter Queues (DLQs) are a powerful feature of Amazon Simple Queue Service (SQS) that allows you to isolate and handle failed messages. Messages are sent to the DLQ for further analysis and troubleshooting when they fail to be processed successfully. AWS provides an automated way to redrive messages from the standard queues but as of today there is no such option for the FIFO queue.
Having that said and because of the need to gain some robustness in my day-to-day life as DevOps Engineer, once the underlying issues are resolved, it's essential to automate the process of redriving these messages back to the original FIFO queue for successful processing avoiding the error-prone approach of human taking manual action on message by message basis.
In this blog post, we'll explore how to automate the Redrive process using an AWS Lambda function and an infrastructure-as-code (IaC) template.

Solution:

Prerequisites:
Before diving into the automation process, make sure you have resolved the root cause that led to messages ending up in the DLQ. Redriving messages without addressing the underlying issues can result in an endless loop of failures. It's important to thoroughly test and validate your application code to ensure it can successfully process the redriven messages.

Infrastructure as Code (IaC) Template:
We'll utilize an AWS CloudFormation template written in YAML to deploy the Lambda in an IaC way. This template provisions an AWS Lambda function that reads all the messages from the DLQ and sends them back to the original FIFO queue. Here's the IaC template:

AWSTemplateFormatVersion: 2010-09-09
Description: Automate Redrive from DLQ for FIFO SQS queue

Transform:
- AWS::Serverless-2016-10-31

Resources:

  LambdaFunction:
    Type: AWS::Serverless::Function
    Properties:
      Handler: src/lambda_function.handler
      Runtime: python3.10
      Architectures:
        - x86_64
      MemorySize: 128
      Timeout: 300
      Description: A Lambda function that reads all the SQS messages from DLQ and sends them back to the original FIFO queue. NB! Ensure you have resolved the issue which caused messages to end up in DLQ before running this function.
      Policies:
        - AWSLambdaBasicExecutionRole
        - SQSPollerPolicy:
            QueueName: '*'
        - SQSSendMessagePolicy:
            QueueName: '*'
        -
          Version:  '2012-10-17'
          Statement:
            Effect: Allow
            Action:
              - sqs:ListQueues
              - sqs:GetQueueUrl
              - sqs:ListDeadLetterSourceQueues
              - sqs:ListQueueTags
            Resource: !Sub arn:aws:sqs:${AWS::Region}:${AWS::AccountId}:*

This CloudFormation template defines a serverless AWS Lambda function that will be responsible for reading messages from the DLQ and sending them back to the original FIFO queue.

In order to deploy the SAM template, you will need to execute the following commands

sam build
sam deploy --guided

Application Code:
To implement the functionality of the Lambda function, we'll use Python and the AWS SDK, Boto3. Here's the code:

import boto3

def handler(event, context):
    sqs = boto3.client('sqs')

    dlq_url = event['dlq_name']
    target_queue_url = event['target_queue_name']

    while True:
        response = sqs.receive_message(
            QueueUrl=dlq_url,
            MaxNumberOfMessages=10,
            AttributeNames=['MessageGroupId']
        )

        if 'Messages' not in response:
            print ('No Messages in DLQ')
            break

        for message in response['Messages']:
            response = sqs.send_message(
                QueueUrl=target_queue_url,
                MessageBody=message['Body'],
                MessageGroupId=message['Attributes']['MessageGroupId'],
                MessageDeduplicationId=message['MessageId']
            )

            if 'MessageId' in response:
                # Message redriven successfully, delete it from DLQ
                delete_response = sqs.delete_message(
                    QueueUrl=dlq_url,
                    ReceiptHandle=message['ReceiptHandle']
                )
                if 'ResponseMetadata' in delete_response and delete_response['ResponseMetadata']['HTTPStatusCode'] != 200:
                    raise Exception('Failed to delete message from DLQ: {}'.format(delete_response))
            else:
                # Handle error case when redriving message fails
                print('Failed to redrive message: {}'.format(response))

    return {
        'statusCode': 200,
        'body': 'DLQ messages redriven successfully.'
    }

This code defines the Lambda function's handler, which takes the event and context as input parameters. The function utilizes the Boto3 SQS client to retrieve messages from the DLQ (dlq_url) and sends them back to the original FIFO queue (target_queue_url). It also handles the deletion of successfully redriven messages and reports any failures.

Deployment and Configuration:
To deploy the solution, you can use the SAM template provided. Customize the template parameters according to your requirements, such as the DLQ name and the target FIFO queue name. Once deployed, the Lambda function could be triggered on-demand to process messages from the DLQ with the following payload:

{
    "dlq_name": "my-sqs-queue-dlq.fifo",
    "target_queue_name": "my-sqs-queue.fifo"
}

NB! You need to change the values above as appropriate to your case.

To ensure the successful execution of the Lambda function, make sure the IAM role associated with the function has appropriate permissions. The provided template includes the necessary policies for accessing the SQS queues and executing Lambda functions.

Monitoring and Error Handling:
It's crucial to monitor the execution of the Lambda function and handle any potential errors. You can enable logging for the Lambda function and configure a CloudWatch Log Group to capture logs. Monitor the logs for any failures during the redriving process and take appropriate action to rectify the issues.

Additionally, consider setting up alarms and notifications using Amazon CloudWatch. Configure alarms to trigger notifications when the Redrive function encounters errors or exceeds certain thresholds, ensuring timely intervention and troubleshooting.

Conclusion:
Automating the Redrive process from the Dead Letter Queue for a FIFO SQS queue is crucial to ensure the successful processing of failed messages. By using AWS Lambda and the provided infrastructure as code template, you can set up a scalable and reliable solution. Remember to address the root cause of message failures before initiating the Redrive process.

By leveraging the power of AWS services and automation, you can enhance the resiliency and reliability of your message-processing workflows. With proper monitoring and error handling, you can ensure the smooth and efficient operation of your FIFO SQS queues.

Spinning up EC2 instance part of ASG faster - mission impossible? Or maybe not?!

Todor Todorov — Mon, 15 Aug 2022 13:46:30 +0000

We all know about all these serverless opportunities out there running on lightweight docker images launched in seconds or Lamba functions triggered in time comparable to lightning speed.
However, even if I would prefer and I am a fan of all these serverless solutions like ECS Fargate, Lambda, EKS, etc., it is more than obvious that this could not be a panacea and could not solve all the cases with serverless.
Similarly to every technology EC2 also does not make an exception and does have advantages and things to consider as bottlenecks too.
EC2 in its essence represents a virtual machine where we have a full-sized Operating System and could not escape from the boot time factor. In the meantime, AWS is a leader which pays attention to what its customers' needs are.
This is why back in 2019, a decade after EBS appeared, they presented a feature called EBS Fast Snapshot Restore (FSR). This feature enables us to use a snapshot and create a fresh EBS volume with up to 16 TiB space and 64K IOPS.
As you may already know, the main benefit of this feature is to make the EC2 boot faster.

Use Case

Speed up the boot process of a new instance part of an AutoScaling Group in a cost-effective way.

Question

Could I benefit from the FSR feature and at the same time avoid any potential extra cost?

Solution

Fast Snapshot Restore feature is not free of charge hence we need to handle with care its usage just like we do with all the other services which power our business. To do that we should first know that it will charge us $0.75 for each hour that Fast Snapshot Restore is enabled for a snapshot in a particular Availability Zone. The second thing which we need to know is when we can benefit from the feature and the answer is simple - during the boot time only (once the instance is operational, we can disable it till the next time we need it). So far so good, but outages come with no upfront notice and we should be ready, this is why some automation will give us a hand in this situation. If you are using ASG, you are most probably aware of the existence of Lifecycle hooks or in other words - the opportunity to halt the launching/termination of an EC2 instance and trigger some actions before you let it go terminating or launching.
Here is a small diagram that will make the situation clear:

How setup will look like in CloudForamtion Template to deploy the Step Function is shown below:

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: Enable/Disable FRS on newly launched instance in AutoScaling Group

Parameters:

  Asg:
    Description: 'Source ASG from where to get an instance Id to create the new AMI'
    Type: String
    Default: MyASG

  LambdaManageFrsZipLocation:
    Description: 'Name of the Python Lambda function to manage AMIs zip file'
    Type: String
    Default: 'FRS.zip'

  LambdaStartStepFunctionZipLocation:
    Description: 'Name of the Python Lambda function to start step function zip file'
    Type: String
    Default: 'StartStepFunction.zip'

  LambdaCompleteLifecycleHookZipLocation:
    Description: 'Name of the Python Lambda function to complete lifecycle hook zip file'
    Type: String
    Default: 'CompleteLifecycleHook.zip'

  Region:
    Description: 'Operational Region'
    Type: String
    Default: 'us-east-1'

  Environment:
    Description: 'Environment Name'
    Type: String
    Default: 'Prod'

  ActivateLifecycleWhenLaunchOrch:
    Description: 'Has the FRS to be activated when the instance is launching (Y/N) ?'
    Type: String
    Default: Y
    AllowedValues:
      - Y
      - N

#---------------------------------------------------------------------------------------------------------------------------------------------------------
Conditions:
  ActivateLifecycleConditionStart: !Equals [!Ref 'ActivateLifecycleWhenLaunchOrch', 'Y']
#---------------------------------------------------------------------------------------------------------------------------------------------------------

Resources:
  myStepFunctionRole:
    Type: 'AWS::IAM::Role'
    Properties:
      RoleName: !Sub 'StepFunctionRole-${Environment}-${AWS::Region}'
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service: states.amazonaws.com
            Action: 'sts:AssumeRole'
      Path: /
      Policies:
        - PolicyName: ManageAmiStepFunctionPolicy
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Sid: VisualEditor0
                Effect: Allow
                Action:
                  - 'logs:CreateLogStream'
                  - 'logs:PutLogEvents'
                Resource: 'arn:aws:logs:*:*:*'
              - Sid: VisualEditor1
                Effect: Allow
                Action:
                  - 'lambda:InvokeFunction'
                Resource: '*'
              - Sid: SomeNewSid
                Effect: Allow
                Action: 'logs:CreateLogGroup'
                Resource: '*'

  myLambdaManageAmiRole:
    Type: AWS::IAM::Role
    Properties:
      Description: 'Manage AMI'
      MaxSessionDuration: 3600
      Path: '/service-role/'
      RoleName: !Sub 'ManageFrsLambdaRole_${Environment}'
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
        - Effect: Allow
          Principal:
            Service:
            - lambda.amazonaws.com
          Action:
          - sts:AssumeRole
      Policies:
        - PolicyName: allowLogging
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
            - Effect: Allow
              Action:
              - logs:CreateLogGroup
              - logs:CreateLogStream
              - logs:PutLogEvents
              Resource: '*'
        - PolicyName: allowEC2
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
            - Effect: Allow
              Action:
              - ec2:DescribeImages
              - ec2:CreateTags
              - ec2:DescribeFastSnapshotRestores
              - ec2:DisableFastSnapshotRestores
              - ec2:EnableFastSnapshotRestores
              Resource: '*'
        - PolicyName: allowDescribeAutoScaling
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
            - Effect: Allow
              Action:
              - autoscaling:DescribeAutoScalingGroups
              Resource: '*'
        - PolicyName: allowIamPassRole
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
            - Effect: Allow
              Action:
              - iam:PassRole
              - iam:ListAccessKeys
              Resource: !Sub "arn:aws:iam::${AWS::AccountId}:role/*"
        - PolicyName: allowKMS
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
            - Effect: Allow
              Action:
              - kms:ReEncrypt*
              - kms:GenerateDataKey*
              - kms:CreateGrant
              - kms:DescribeKey*
              - kms:ListKeys
              - kms:ListAliases
              Resource: '*'
        - PolicyName: allowSSMCommands
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
            - Effect: Allow
              Action:
              - ssm:SendCommand
              - ssm:GetCommandInvocation
              Resource: '*'

  myLambdatoStartStepFunctionRole:
    Type: AWS::IAM::Role
    Properties:
      Description: 'Start Step Function'
      MaxSessionDuration: 3600
      Path: '/service-role/'
      RoleName: !Sub 'StartStepFunctionLambdaRole_${Environment}'
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
        - Effect: Allow
          Principal:
            Service:
            - lambda.amazonaws.com
          Action:
          - sts:AssumeRole
      Policies:
        - PolicyName: allowLogging
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
            - Effect: Allow
              Action:
              - logs:CreateLogGroup
              - logs:CreateLogStream
              - logs:PutLogEvents
              Resource: '*'
        - PolicyName: allowStates
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
            - Effect: Allow
              Action:
              - states:StartExecution
              Resource: 
              - !GetAtt myFastRestoreStepFunction.Arn
        - PolicyName: allowSNS
          PolicyDocument:
            Version: 2012-10-17
            Statement:
            - Effect: Allow
              Resource: "*"
              Action:
              - sns:Publish
        - PolicyName: allowASG
          PolicyDocument:
            Version: 2012-10-17
            Statement:
            - Effect: Allow
              Resource: "*"
              Action:
              - autoscaling:CompleteLifecycleAction

  myLifecycleHookRole:
    Type: "AWS::IAM::Role"
    Properties:
        AssumeRolePolicyDocument:
            Version: "2012-10-17"
            Statement:
              -
                Effect: "Allow"
                Action:
                  - "sts:AssumeRole"
                Principal:
                    Service:
                      - "autoscaling.amazonaws.com"
        Path: /
        ManagedPolicyArns:
          - arn:aws:iam::aws:policy/service-role/AutoScalingNotificationAccessRole

  myLambdatoCompleteLifecycleHookRole:
    Type: AWS::IAM::Role
    Properties:
      Description: 'Complete Lifecycle Hook'
      MaxSessionDuration: 3600
      Path: '/service-role/'
      RoleName: !Sub 'CompleteLifecycleHookLambdaRole_${Environment}'
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
        - Effect: Allow
          Principal:
            Service:
            - lambda.amazonaws.com
          Action:
          - sts:AssumeRole
      Policies:
        - PolicyName: allowLogging
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
            - Effect: Allow
              Action:
              - logs:CreateLogGroup
              - logs:CreateLogStream
              - logs:PutLogEvents
              Resource: '*'

        - PolicyName: allowASG
          PolicyDocument:
            Version: 2012-10-17
            Statement:
            - Effect: Allow
              Resource: "*"
              Action:
              - autoscaling:CompleteLifecycleAction

  myLambdatoCompleteLifecycleHook:
    Type: AWS::Lambda::Function
    Properties:
      Code:
        S3Bucket: !Sub 'cft-templates.${AWS::AccountId}.${AWS::Region}'
        S3Key: !Ref LambdaCompleteLifecycleHookZipLocation
      Description: 'Complete Lifecycle Hook'
      FunctionName: !Sub 'CompleteLifecycleHookLambda_${Environment}'
      Handler: CompleteLifecycleHook.lambda_handler
      Role: !GetAtt myLambdatoCompleteLifecycleHookRole.Arn
      Runtime: 'python3.8'
      Timeout: 300

  myFastRestoreStepFunction:
    Type: AWS::Serverless::StateMachine
    Properties:
      Name: !Sub 'ManageFastRestoreStepFunction_${Environment}'
      Type: STANDARD
      Role: !GetAtt myStepFunctionRole.Arn
      Definition:
        #Comment: Step Function to Enable FastRestore
        StartAt: Search for EC2 Id in ASG #Started by Lifecycle Hook 1 ?
        States:

          Search for EC2 Id in ASG:
            Type: Task
            OutputPath: '$.Payload'
            Resource: arn:aws:states:::lambda:invoke
            Parameters:
              FunctionName: !GetAtt myLambdatoManageFrs.Arn
              Payload:
                ASGName.$: '$.ASGName'
                lifeCycleHook.$: '$.lifeCycleHook'
                instance.$: '$.instance'
                region.$: '$.region'
                status.$: '$.status'
                AMIid.$: '$.AMIid'
                cmd: 'getASGInstanceId'
            Next: Get AMI ID

          Get AMI ID:
            Type: Task
            OutputPath: '$.Payload'
            Resource: arn:aws:states:::lambda:invoke
            Parameters:
              FunctionName: !GetAtt myLambdatoManageFrs.Arn
              Payload:
                ASGName.$: '$.ASGName'
                lifeCycleHook.$: '$.lifeCycleHook'
                instance.$: '$.instance'
                region.$: '$.region'
                status.$: '$.status'
                AMIid.$: '$.AMIid'
                cmd: 'getInstanceImageId'
            Next: Enable FastSnapshotRestores

          Enable FastSnapshotRestores:
            Type: Task
            OutputPath: '$.Payload'
            Resource: arn:aws:states:::lambda:invoke
            Parameters:
              FunctionName: !GetAtt myLambdatoManageFrs.Arn
              Payload:
                ASGName.$: '$.ASGName'
                lifeCycleHook.$: '$.lifeCycleHook'
                instance.$: '$.instance'
                region.$: '$.region'
                status.$: '$.status'
                AMIid.$: '$.AMIid'
                cmd: 'enableFastRestore'
            Next: Check FastRestore

          Test FastRestore:
            Type: Choice
            Choices:
            - Variable: '$.status'
              StringEquals: enabled
              Next: Parallel
            - Not:
                Variable: '$.status'
                StringEquals: enabled
              Next: Wait 20s Fast

          Wait 20s Fast:
            Type: Wait
            Seconds: 20
            Next: Check FastRestore

          Check FastRestore:
            Type: Task
            OutputPath: '$.Payload'
            Resource: arn:aws:states:::lambda:invoke
            Parameters:
              FunctionName: !GetAtt myLambdatoManageFrs.Arn
              Payload:
                ASGName.$: '$.ASGName'
                lifeCycleHook.$: '$.lifeCycleHook'
                instance.$: '$.instance'
                region.$: '$.region'
                status.$: '$.status'
                AMIid.$: '$.AMIid'
                cmd: 'showFastRestore'
            Next: Test FastRestore
          Parallel:
            Type: Parallel
            Branches:
              [
              {
              "StartAt": "Wait for Disable FastRestore",
              "States": {
                "Wait for Disable FastRestore": {
                  "Seconds": 3000,
                  "Type": "Wait",
                  "Next": "Disable FastSnapshotRestores"
                },
                "Disable FastSnapshotRestores": {
                  "Next": "success",
                  "OutputPath": "$.Payload",
                  "Parameters": {
                    "FunctionName": !GetAtt myLambdatoManageFrs.Arn,
                    "Payload": {
                      "AMIid.$": "$.AMIid",
                      "ASGName.$": "$.ASGName",
                      "cmd": "disableFastRestore",
                      "instance.$": "$.instance",
                      "lifeCycleHook.$": "$.lifeCycleHook",
                      "region.$": "$.region",
                      "status.$": "$.status"

                    }
                  },
                  "Resource": "arn:aws:states:::lambda:invoke",
                  "Type": "Task"
                },
                "success": {
                  "Type": "Pass",
                  "End": true
                }
              }
              },
              {
              "StartAt": "Complete Lifecycle Hook",
              "States": {
                "Complete Lifecycle Hook": {
                  "OutputPath": "$.Payload",
                  "Parameters": {
                    "FunctionName": !GetAtt myLambdatoCompleteLifecycleHook.Arn,
                    "Payload": {
                      "AMIid": "null",
                      "ASGName.$": "$.ASGName",
                      "cmd": "null",
                      "instance.$": "$.instance",
                      "lifeCycleHook.$": "$.lifeCycleHook",
                      "region": "null",
                      "status": "null"
                    }
                  },
                  "Resource": "arn:aws:states:::lambda:invoke",
                  "Type": "Task",
                  "Catch": [
                    {
                      "ErrorEquals": [
                        "States.ALL"
                      ],
                      "Next": "handle failure",
                      "ResultPath": "$.error"
                    }
                  ],
                  "End": true
                },
                "handle failure": {
                  "Type": "Pass",
                  "End": true
                }
              }
              }
              ]
            "End": true


  #Lambda to trigger the Step Function dedicated to enable FastRestore Snapshot on new EC2 instance launch in the ASG
  myLambdatoStartFastRestoreStepFunction:
    Type: AWS::Lambda::Function
    Properties:
      Environment:
        Variables:
          STATE_MACHINE_ARN: !GetAtt myFastRestoreStepFunction.Arn
      Code:
        S3Bucket: !Sub 'cft-templates.${AWS::AccountId}.${AWS::Region}'
        S3Key: !Ref LambdaStartStepFunctionZipLocation
      Description: 'Launch Step Function handling FRS enable/disable on EC2 launch'
      FunctionName: !Sub 'StartFastRestoreStepFunctionLambda_${Environment}'
      Handler: StartStepFunction.lambda_handler
      Role: !GetAtt myLambdatoStartStepFunctionRole.Arn
      Runtime: 'python3.8'
      Timeout: 300

  # Lifecycle hook to trigger the Enable/Disable of FRS when instance is launched on the Orchestrator ASG
  myStartLifecycleHookTopic:
    Type: AWS::SNS::Topic

  myStartLifecycleHookSubscription:
    Type: AWS::SNS::Subscription
    Properties:
      Endpoint: !GetAtt myLambdatoStartFastRestoreStepFunction.Arn
      Protocol: "lambda"
      TopicArn: !Ref myStartLifecycleHookTopic

  myStartPermission:
    Type: AWS::Lambda::Permission
    Properties:
      Action: "lambda:InvokeFunction"
      FunctionName: !GetAtt myLambdatoStartFastRestoreStepFunction.Arn
      Principal: sns.amazonaws.com
      SourceArn: !Ref myStartLifecycleHookTopic

  myStartLifecycleHookASG:
    Type: AWS::AutoScaling::LifecycleHook
    Condition: ActivateLifecycleConditionStart
    Properties:
      AutoScalingGroupName: !Ref Asg
      LifecycleTransition: "autoscaling:EC2_INSTANCE_LAUNCHING"
      DefaultResult: CONTINUE
      HeartbeatTimeout: 600
      NotificationMetadata: !Sub |-
        {
          "lifeCycleHook": "null",
          "ASGName": "${Asg}",
          "instance": "null",
          "region": "${Region}",
          "status": "null",
          "AMIid": "null",
          "cmd": "null"
        }
      NotificationTargetARN:
        Ref: myStartLifecycleHookTopic
      RoleARN: !GetAtt myLifecycleHookRole.Arn

  myLambdatoManageFrs:
    Type: AWS::Lambda::Function
    Properties:
      Code:
        S3Bucket: !Sub 'cft-templates.${AWS::AccountId}.${AWS::Region}'
        S3Key: !Ref LambdaManageFrsZipLocation
      Description: 'Manage Fast Restore Snapshot'
      FunctionName: !Sub 'ManageFRS_${Environment}'
      Handler: FRS.lambda_handler
      Role: !GetAtt myLambdaManageFrsRole.Arn
      Runtime: 'python3.8'
      Timeout: 300

  myLambdaManageFrsRole:
    Type: AWS::IAM::Role
    Properties:
      Description: 'Manage FRS'
      MaxSessionDuration: 3600
      Path: '/service-role/'
      RoleName: !Sub 'ManageFRS_${Environment}'
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
        - Effect: Allow
          Principal:
            Service:
            - lambda.amazonaws.com
          Action:
          - sts:AssumeRole
      Policies:
        - PolicyName: allowLogging
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
            - Effect: Allow
              Action:
              - logs:CreateLogGroup
              - logs:CreateLogStream
              - logs:PutLogEvents
              Resource: '*'
        - PolicyName: allowEC2
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
            - Effect: Allow
              Action:
              - ec2:CreateTags
              - ec2:Describe*
              - ec2:DisableFastSnapshotRestores
              - ec2:EnableFastSnapshotRestores
              Resource: '*'
        - PolicyName: allowDescribeAutoScaling
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
            - Effect: Allow
              Action:
              - autoscaling:DescribeAutoScalingGroups
              Resource: '*'
        - PolicyName: allowIamPassRole
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
            - Effect: Allow
              Action:
              - iam:PassRole
              - iam:ListAccessKeys
              Resource: !Sub "arn:aws:iam::${AWS::AccountId}:role/*"

We will have several Lambda Functions in order to cover the logic:

FRS Lambda:

import boto3
from botocore.config import Config
import json
from datetime import tzinfo, timedelta, datetime
import time
import sys
import os

# --------------------------------------------------------------------------------------------------------------------------------
# To display debug messages (True or False)
DEBUG = True

def message(msg):
    if DEBUG:
        print(msg)
    return

def error(err, msg ):
    raise Exception (f"{msg} --- err = {err}")

def getASGInstanceId(asgClient, asgName):
    try:
        message("---- getInstanceId")

        # get object for the ASG we're going to update, filter by name of target ASG
        response = asgClient.describe_auto_scaling_groups(AutoScalingGroupNames=[asgName])

        if not response['AutoScalingGroups']:
            err = f"## No such ASG '{asgName}'"
            message(err)
            raise RuntimeError(err)

        # get InstanceID in current ASG that we'll use to model new Launch Configuration after
        instanceId = response.get('AutoScalingGroups')[0]['Instances'][-1]['InstanceId']

    except Exception as e:
        message(f"==== e = {e}")
        return(error(e, f"## Failed to get ASG Instance Id of ASG '{asgName}'"))

    message(f"InstanceId = {instanceId}")
    return instanceId
def getInstanceImageId(instanceId):
    ec2_client = boto3.client('ec2')
    ec2_response = ec2_client.describe_instances(
    InstanceIds = [instanceId]
      )
    for instances in ec2_response['Reservations']:
        for image in instances['Instances']:
            print(image['ImageId'])
            AMIid = image['ImageId']
    return AMIid
def enableFastRestore(AMIid,asgName):
    ec2 = boto3.client('ec2')
    ec2response = ec2.describe_images(
        ImageIds=[
            AMIid
        ],
    )

    asg  = boto3.client('autoscaling')
    # You may need to edit the filter here for your use case
    for i  in range(5):
        if ec2response['Images'][0]['BlockDeviceMappings'][i]['Ebs']['VolumeSize'] < 40:
            snapshotid=ec2response['Images'][0]['BlockDeviceMappings'][i]['Ebs']['SnapshotId']
    print(snapshotid)
    asgresponse = asg.describe_auto_scaling_groups(
                AutoScalingGroupNames=[
                            asgName,
                                ]
                        )

    print(asgresponse['AutoScalingGroups'][0]['AvailabilityZones'][0])
    az=asgresponse['AutoScalingGroups'][0]['AvailabilityZones'][0]
    try:
        fsrresponse = ec2.enable_fast_snapshot_restores(
                   AvailabilityZones=[
                         az,
                   ],
                   SourceSnapshotIds=[
                         snapshotid,
                   ]
        )
    except Exception as e:
        message(f"==== e = {e}")
        return(error(e, f"## Failed to Enable Fast Restore "))
    return fsrresponse

# deactivate FastRestore Snapshot

def disableFastRestore(AMIid,asgName):
    ec2 = boto3.client('ec2')
    ec2response = ec2.describe_images(
        ImageIds=[
            AMIid
        ],
    )

    asg  = boto3.client('autoscaling')
    # You may need to edit the filter here for your use case
    for i  in range(5):
        if ec2response['Images'][0]['BlockDeviceMappings'][i]['Ebs']['VolumeSize'] < 40:
            snapshotid=ec2response['Images'][0]['BlockDeviceMappings'][i]['Ebs']['SnapshotId']
    print(snapshotid)
    asgresponse = asg.describe_auto_scaling_groups(
                AutoScalingGroupNames=[
                            asgName,
                                ]
                        )

    print(asgresponse['AutoScalingGroups'][0]['AvailabilityZones'][0])
    az=asgresponse['AutoScalingGroups'][0]['AvailabilityZones'][0]
    try:
        fsrresponse = ec2.disable_fast_snapshot_restores(
                   AvailabilityZones=[
                         az,
                   ],
                   SourceSnapshotIds=[
                         snapshotid,
                   ]
        )
    except Exception as e:
        message(f"==== e = {e}")
        return(error(e, f"## Failed to Disable Fast Restore "))
    return fsrresponse


def showFastRestore(AMIid,asgName):
    ec2 = boto3.client('ec2')
    ec2response = ec2.describe_images(
        ImageIds=[
            AMIid
        ],
    )

    asg  = boto3.client('autoscaling')
    # You may need to edit filter here for your use case
    for i  in range(5):
        if ec2response['Images'][0]['BlockDeviceMappings'][i]['Ebs']['VolumeSize'] < 40:
            snapshotid=ec2response['Images'][0]['BlockDeviceMappings'][i]['Ebs']['SnapshotId']
    print(snapshotid)
    asgresponse = asg.describe_auto_scaling_groups(
                AutoScalingGroupNames=[
                            asgName,
                                ]
                        )

    print(asgresponse['AutoScalingGroups'][0]['AvailabilityZones'][0])
    az=asgresponse['AutoScalingGroups'][0]['AvailabilityZones'][0]
    try:
        fsrresponse = ec2.describe_fast_snapshot_restores(
            Filters=[
               {
                'Name': 'snapshot-id',
                'Values': [
                             snapshotid,
                       ]
                    },
            ]
        )
    except Exception as e:
        message(f"==== e = {e}")
        return(error(e, f"## Failed to Disable Fast Restore "))
    fsrstate=fsrresponse['FastSnapshotRestores'][0]['State']
    return fsrstate

def lambda_handler(event, context):
    message("Received event: " + json.dumps(event, indent=2))
#----------------
    lifeCycleHook = event['lifeCycleHook']
    asgName = event['ASGName']
    instance = event['instance']
    status = event['status']
    region_name=event['region']
    status=event['status']
    AMIid=event['AMIid']
    cmd = event['cmd']

#----------------
    autoscalingClient = boto3.client('autoscaling')

    ec2Client   = boto3.client('ec2', region_name)

    ec2Resource   = boto3.resource('ec2', region_name)

    ssmClient   = boto3.client('ssm',  region_name)

#----------------

    message(f"---- CMD = {cmd}")

    if cmd == 'getASGInstanceId':
        message("---- Get instance Id from ASG")

        instance = getASGInstanceId(autoscalingClient, asgName)
        AMIid = getInstanceImageId(instance)
    elif cmd == 'getInstanceImageId':
        instance = getASGInstanceId(autoscalingClient, asgName)
        message("---- Get Instance Image Id on instance: " + instance)
        AMIid = getInstanceImageId(instance)
    elif cmd == 'enableFastRestore':
        message("---- Enable Fast Restore")
        instance = getASGInstanceId(autoscalingClient, asgName)
        AMIid = getInstanceImageId(instance)
        response = enableFastRestore(AMIid,asgName)
    elif cmd == 'disableFastRestore':
        instance = getASGInstanceId(autoscalingClient, asgName)
        message("---- Disable Fast Restore")
        AMIid = getInstanceImageId(instance)
        response = disableFastRestore(AMIid,asgName)
    elif cmd == 'showFastRestore':
        message("---- Show Fast Restore")
        instance = getASGInstanceId(autoscalingClient, asgName)
        AMIid = getInstanceImageId(instance)
        status = showFastRestore(AMIid,asgName)

    else:
        return(error(e, f"Command '{cmd}' is not recognized or planned to be managed" ))

    result = {"lifeCycleHook": lifeCycleHook, "ASGName": asgName, "instance": instance,  "region": region_name, "status": status,  "AMIid": AMIid, "cmd": cmd}
    message("##### RETURN result :  " + json.dumps(result, indent=2))
    return result

Complete Lifecycle Hook Lambda:

import boto3
from botocore.exceptions import ClientError
import os
import json


asgClient = boto3.client('autoscaling')

def lambda_handler(event, context):
    print('The Lambda function is starting.')
    print("Received event: " + json.dumps(event, indent=2))

    autoScalingGroup = event['ASGName']
    instanceId = event['instance']
    lifeCycleHook = event['lifeCycleHook']

    actionResult = "CONTINUE"

    response = asgClient.complete_lifecycle_action(
        LifecycleHookName = lifeCycleHook,
        AutoScalingGroupName = autoScalingGroup,
        LifecycleActionResult = actionResult,
        InstanceId = instanceId
    )

    print(f"Complete lifecycle hook response : {response}")
    return

Start Step Function Lambda:

import boto3
from datetime import tzinfo, timedelta, datetime
from botocore.exceptions import ClientError
import os
import json
import ast

STATE_MACHINE_ARN = os.environ.get('STATE_MACHINE_ARN')

stepFnClient = boto3.client('stepfunctions')

def lambda_handler(event, context):
    print('The Lambda function is starting.')
    print("Received event: " + json.dumps(event, indent=2))

    datename = datetime.now().strftime('%Y_%m_%d_%HH%M')
    EXECUTION_NAME = 'Life_Cycle_Hook_Start_' + datename
    message = event['Records'][0]['Sns']['Message']
    msgJson = json.loads(message)
    instanceId=msgJson["EC2InstanceId"]
    lifecycleHookName=msgJson["LifecycleHookName"]
    lifecycleActionToken=msgJson["LifecycleActionToken"]
    autoScalingGroupName=msgJson["AutoScalingGroupName"]
    notificationMetadata=msgJson["NotificationMetadata"]
    notificationMetadataDict = json.loads(notificationMetadata)

    print(f"----- NotificationMetadataDict = {notificationMetadataDict}")
    notificationMetadataDict["lifeCycleHook"] = lifecycleHookName
    notificationMetadataDict["instance"] = instanceId
    notificationMetadataDict["ASGName"] = autoScalingGroupName

    print(f"----- NotificationMetadataDict = {notificationMetadataDict}")
    notificationMetadata = json.dumps(ast.literal_eval(str(notificationMetadataDict)), indent=2)
    print(f"----- NotificationMetadata = {notificationMetadata}")

    print('Starting step function ...')

    response = stepFnClient.start_execution(
        stateMachineArn=STATE_MACHINE_ARN,
        name=EXECUTION_NAME,
        input=notificationMetadata
    )

    print(f"Execution arn of the Step Function : {response.get('executionArn')}")
    return

To Sum up - I have tried to present you with an approach that will allow you to benefit from Fast Snapshot Restore functionality and in the meantime reduce the spending on the service to the really needed one.

How to sync CloudFormation templates from CodeCommit to S3

Todor Todorov — Tue, 21 Jun 2022 21:44:12 +0000

If it happens to you to work on a project where you have selected for one or another reason CloudFormation, you most probably have fallen into a situation to ask yourself the same question as me "OK! But how could I reduce the manual effort and risk from human error while maintaining my CloudFormation Templates between CodeCommit and S3?".
You can easily replace CodeCommit with any other SCM platform, however, this topic will be still valid.

CloudFormation is the managed service offered by AWS for managing your infrastructure as a code. Whether you prefer JSON or YAML files you will always need a location to centrally store your templates.

Use case

Keep track of the changes made to the files when in the meantime you do not need to sync templates to the S3 bucket artifact location manually which will increase the risk of human error during the manual synchronization.

Question

How can we automate this and always have near-real-time synchronization between our repo and S3 bucket from which we point URLs inside the CloudFormation stacks?

Solution

As with every topic we need to cover, the first thing to do is what?
Diagram of course ...

Here is a sample of a CloudFormation template that we can utilize to achieve the goal.
A prerequisite is to have a CodeCommit repository created in advance.

AWSTemplateFormatVersion: 2010-09-09
Parameters:
  pCodeCommitRepo:
    Description: This parameter serves to point which is our repository to use as a single point of truth
  pSolutionName:
    Description: Name which we can use to spin up more than one Pipeline for different repositories
  pBucketName:
    Type: String
    Default: my-cf-templates
  pMyAWSKeyAccountId:
    Type: String
    Default: 123456789012
  pMyAWSKeyRegion:
    Type: String
    Default: eu-west-1
  pMyAWSKeyArn:
    Type: String
    Default: arn:aws:kms:eu-west-1:123456789012:key/abcdefgh-ijkl-1234-5678-zxcvbnmasdfg

Resources:
  rS3Bucket:
    Type: AWS::S3::Bucket
    DeletionPolicy: Retain
    Properties:
      BucketName: !Ref pBucketName
      VersioningConfiguration:
        Status: Enabled
      PublicAccessBlockConfiguration:
          BlockPublicAcls: true
          BlockPublicPolicy: true
          IgnorePublicAcls: true
          RestrictPublicBuckets: true
      BucketEncryption:
        ServerSideEncryptionConfiguration:
          - ServerSideEncryptionByDefault:
              SSEAlgorithm: 'aws:kms'
              KMSMasterKeyID: !GetAtt rKMSS3Encryption.Arn
      # What we can do here, if we have multiple AWS accounts where we are going to use the CloudFormation Templates is to replicate
      # the content in a dedicated S3 bucket inside every AWS account where we need.
      ReplicationConfiguration:
          Role: !GetAtt rS3ReplicationRole.Arn
          Rules:
            - Destination:
                Bucket: !Sub "arn:aws:s3:::my-cf-templates-${pMyAWSKeyAccountId}.${pMyAWSKeyRegion}"
                AccessControlTranslation:
                  Owner: Destination
                Account: !Ref pMyAWSKeyAccountId
                EncryptionConfiguration:
                  ReplicaKmsKeyID: !Ref pMyAWSKeyArn
              DeleteMarkerReplication:
                Status: Enabled
              Priority: 1
              Filter:
                Prefix: ''
              Status: Enabled
              SourceSelectionCriteria:
                SseKmsEncryptedObjects:
                  Status: Enabled

  rKMSS3Encryption:
    Type: AWS::KMS::Key
    DeletionPolicy: Retain
    Properties:
      Enabled: true
      KeyPolicy:
        Version: '2012-10-17'
        Id: key-default-1
        Statement:
          - Sid: Enable IAM User Permissions
            Effect: Allow
            Principal:
              AWS:
                Fn::Join:
                - ''
                - - 'arn:aws:iam::'
                  - Ref: AWS::AccountId
                  - :root
            Action: kms:*
            Resource: '*'
          - Sid: Allow  use of the CMK
            Effect: Allow
            Principal:
              AWS:
                - !Sub arn:aws:iam::${AWS::AccountId}:role/service-role/AWSCodePipelineServiceRole-*
            Action:
                - kms:Encrypt
                - kms:Decrypt
                - kms:ReEncrypt*
                - kms:GenerateDataKey*
                - kms:DescribeKey
            Resource: "*"

      EnableKeyRotation: true

  rKMSS3EncryptionAlias:
      Type: AWS::KMS::Alias
      DeletionPolicy: Retain
      Properties:
        AliasName: alias/myS3BucketKey
        TargetKeyId: !Ref rKMSS3Encryption

  rS3ReplicationPolicy:
    Type: AWS::IAM::ManagedPolicy
    Properties:
      ManagedPolicyName: !Sub myReplicationPolicy-${pBucketName}-${AWS::Region}
      Description: S3 replication policy
      PolicyDocument:
        Version: '2012-10-17'
        Statement:
          -
            Effect: Allow
            Action:
              - 's3:GetReplicationConfiguration'
              - 's3:ListBucket'
            Resource: !Sub arn:aws:s3:::${pBucketName}
          -
            Effect: Allow
            Action:
              - 's3:GetObject*'
            Resource: !Sub arn:aws:s3:::${pBucketName}/*
          -
            Effect: Allow
            Action:
              - 's3:ReplicateObject'
              - 's3:ReplicateDelete'
              - 's3:ReplicateTags'
              - 's3:ObjectOwnerOverrideToBucketOwner'
            Resource: !Sub arn:aws:s3:::my-cf-templates-${pMyAWSKeyAccountId}.${pMyAWSKeyRegion}/*
          -
            Effect: Allow
            Action:
              - 'kms:*'
            Resource: !GetAtt rKMSS3Encryption.Arn
          -
            Effect: Allow
            Action:
              - 'kms:*'
            Resource: !Ref pMyAWSKeyArn

  rS3ReplicationRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: !Sub ReplicationRole-${pBucketName}-${AWS::Region}
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: 'Allow'
            Principal:
              Service:
                - 's3.amazonaws.com'
            Action:
              - 'sts:AssumeRole'
      ManagedPolicyArns:
        - !Ref rS3ReplicationPolicy

  rCodePipeline:
    Type: AWS::CodePipeline::Pipeline
    Properties:
      Name: !Sub "${pSolutionName}-pipeline"
      RestartExecutionOnUpdate: false
      RoleArn: !GetAtt rCodePipelineServiceRole.Arn
      Stages:
        - Name: Source
          Actions:
            - Name: SourceAction
              ActionTypeId:
                Category: Source
                Owner: AWS
                Provider: CodeCommit
                Version: 1
              Configuration:
                RepositoryName: !Ref pCodeCommitRepo
                BranchName: master
              RunOrder: 1
        - Name: SyncToS3
          Actions:
            - Name: SyncToS3
              ActionTypeId:
                Category: Deploy
                Owner: AWS
                Version: 1
                Provider: CodeDeploy
              Configuration:
                Extract: true
                BucketName: !Ref rS3Bucket
                ObjectKey: "/"
              RunOrder: 1
  rCodePipelineServiceRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Principal:
              Service:
                - codepipeline.amazonaws.com
            Action: 'sts:AssumeRole'
      Path: /
      Policies:
        - PolicyName: !Sub "AWS-CodePipeline-Service-${pCodeCommitRepo}"
          PolicyDocument:
            Version: 2012-10-17
            Statement:
              - Effect: Allow
                Action:
                  - 'codecommit:CancelUploadArchive'
                  - 'codecommit:GetBranch'
                  - 'codecommit:GetCommit'
                  - 'codecommit:GetUploadArchiveStatus'
                  - 'codecommit:UploadArchive'
                Resource: '*'
              - Effect: Allow
                Action:
                  - 'codedeploy:CreateDeployment'
                  - 'codedeploy:GetApplicationRevision'
                  - 'codedeploy:GetDeployment'
                  - 'codedeploy:GetDeploymentConfig'
                  - 'codedeploy:RegisterApplicationRevision'
                Resource: '*'
              - Effect: Allow
                Action:
                  - 'codebuild:BatchGetBuilds'
                  - 'codebuild:StartBuild'
                Resource: '*'
              - Effect: Allow
                Action:
                  - 'devicefarm:ListProjects'
                  - 'devicefarm:ListDevicePools'
                  - 'devicefarm:GetRun'
                  - 'devicefarm:GetUpload'
                  - 'devicefarm:CreateUpload'
                  - 'devicefarm:ScheduleRun'
                Resource: '*'
              - Effect: Allow
                Action:
                  - 'lambda:InvokeFunction'
                  - 'lambda:ListFunctions'
                Resource: '*'
              - Effect: Allow
                Action:
                  - 'iam:PassRole'
                Resource: '*'
              - Effect: Allow
                Action:
                  - 'elasticbeanstalk:*'
                  - 'ec2:*'
                  - 'elasticloadbalancing:*'
                  - 'autoscaling:*'
                  - 'cloudwatch:*'
                  - 's3:*'
                  - 'sns:*'
                  - 'cloudformation:*'
                  - 'rds:*'
                  - 'sqs:*'
                  - 'ecs:*'
                Resource: '*'
  rAmazonCloudWatchEventRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Principal:
              Service:
                - events.amazonaws.com
            Action: 'sts:AssumeRole'
      Path: /
      Policies:
        - PolicyName: !Sub "cw-event-pipeline-execution-${pCodeCommitRepo}"
          PolicyDocument:
            Version: 2012-10-17
            Statement:
              - Effect: Allow
                Action: 'codepipeline:StartPipelineExecution'
                Resource: !Join 
                  - ''
                  - - 'arn:aws:codepipeline:'
                    - !Ref 'AWS::Region'
                    - ':'
                    - !Ref 'AWS::AccountId'
                    - ':'
                    - !Ref rCodePipeline
  rAmazonCloudWatchEventRule:
    Type: AWS::Events::Rule
    Properties:
      EventPattern:
        source:
          - aws.codecommit
        detail-type:
          - CodeCommit Repository State Change
        resources:
          - !Join 
            - ''
            - - 'arn:aws:codecommit:'
              - !Ref 'AWS::Region'
              - ':'
              - !Ref 'AWS::AccountId'
              - ':'
              - !Ref pCodeCommitRepo
        detail:
          event:
            - referenceCreated
            - referenceUpdated
          referenceType:
            - branch
          referenceName:
            - master
      Targets:
        - Arn: !Join 
            - ''
            - - 'arn:aws:codepipeline:'
              - !Ref 'AWS::Region'
              - ':'
              - !Ref 'AWS::AccountId'
              - ':'
              - !Ref rCodePipeline
          RoleArn: !GetAtt rAmazonCloudWatchEventRole.Arn
          Id: codepipeline

In a conclusion, I could say that our target should always be to remove any repetitive effort through automation for the sake of reducing the risk of making errors and at the same time focus on more important and interesting tasks.

DEV Community: Todor Todorov

Proactive RDS Monitoring: How We Built an EventBridge-Powered Alert System for Flawless BFCM Events

The Problem: Silent Database Failures During Peak Traffic

Why Traditional Monitoring Falls Short

The Solution: EventBridge + Lambda + Slack = Proactive Alerts

Architecture Overview

Implementation: Infrastructure as Code

Step 1: Create the Terraform Module

Step 2: Module Variables

Step 3: Use the Module

The Lambda Function: Intelligent Alert Formatting

Complete Lambda Code

Dependencies

Setting Up Slack Integration

What the Slack Alert Looks Like

Benefits: Why This Matters for BFCM

1. Proactive Problem Detection

2. Actionable Alerts

3. Team Collaboration

4. Cost-Effective

5. Scalable Architecture

6. Infrastructure as Code

Real-World Impact

Extending the Solution

Best Practices

Conclusion

Resources

How to Migrate ECR Docker Images Between Repositories (with Automation)

🛠️ Prerequisites

Installing skopeo

📜 The Migration Script

⚡ How It Works

🔧 Example Usage

✅ Why This Helps

🎯 Final Thoughts

(Near)Real-Time Subnet IP Monitoring with AWS Lambda and Slack Alerts

Key Features

Prerequisites

Step 1: Create the SAM Project

Step 2: Implement the Lambda Code

Step 3: Configure the SAM Template

Step 4: Build and Deploy

Step 5: Test and Monitor

Summary

Monitor EBS Volume Health

Prerequisites

Step 1: Create the Project

Step 2: Update the Lambda Code

Step 3: Update the Template

Step 4: Build and Deploy

Step 5: Verify and Monitor

Create Alarms

Summary

Automating Redrive from DLQ for FIFO SQS Queue

The case:

Solution:

Spinning up EC2 instance part of ASG faster - mission impossible? Or maybe not?!

Use Case

Question

Solution

How to sync CloudFormation templates from CodeCommit to S3

Use case

Question

Solution

Installing `skopeo`