DEV Community: vikasbanage

AWS DevOps Agent Multi- Account: Setup and Investigation

vikasbanage — Thu, 30 Jul 2026 04:04:25 +0000

This is the third post in our AWS DevOps Agent series.

Most real applications don't live in a single AWS account — a service might run in one account while its data layer, shared networking, or a sibling microservice sits in another. If AWS DevOps Agent can only see the account it's deployed in, every investigation touching a resource outside that boundary hits a blind spot. Cross-account monitoring closes that gap, letting one Agent Space see across your entire application footprint instead of just its home account.

Prerequisites

Before adding a secondary account, make sure you have:

Access to the AWS DevOps Agent console in the primary account
IAM permissions to create roles in the secondary account

IAM Policies Behind Cross-Account Access

Every secondary account connection rests on three separate policy components, and understanding what each one does makes the setup steps below a lot less mechanical:

Trust policy — lets the AWS DevOps Agent service principal (aidevops.amazonaws.com) assume the role directly. It includes aws:SourceAccount and aws:SourceArn conditions specifically for confused deputy prevention — a security control ensuring only your Agent Space in the primary account can use this trust relationship, not any other AWS DevOps Agent deployment anywhere else.
AIDevOpsAgentAccessPolicy — an AWS-managed policy providing the core read-only permissions the agent needs to investigate resources. AWS maintains and updates this as new capabilities ship.
Inline policy — generated from your specific Agent Space configuration, covering permissions tied to whichever integrations or features you've enabled.

Setting It Up via the Console

Step 1 : Go to respective agent space -> Add Secondary Account -> Select AWS Account

Step 2 : In this we need create IAM roles in secondary account, which will have trust policy from primary account and agent space. Adding account will give you below wizard.

Step 3 :Create the role in the secondary account. In a new browser tab, sign into the secondary account's IAM console → Roles → Create role → Custom trust policy, and paste in the trust policy from Step 2.

Step 4 — Attach the AWS managed policy. Search for and select AIDevOpsAgentAccessPolicy.

Step 5 — Name and create the role. Use the same name you specified in Step 2, add an optional description, and create it.

Step 6 — Attach the inline policy. On the new role's Permissions tab, choose Add permissions → Create inline policy, switch to the JSON tab, and paste the policy the console gave you alongside the trust policy. Name it something identifiable (e.g. DevOpsAgentInlinePolicy) and create it.

Step 7 — Complete the configuration. Back in the primary account's AWS DevOps Agent console, choose Next and confirm the connection status shows Active.

Demo: Seeing Cross-Account Value in Action

The best way to feel why cross-account monitoring matters isn't reading about it — it's watching an investigation hit a wall without it, then watching it succeed with it.

The setup:

Account A (primary, where your Agent Space lives) — a Lambda function and a CloudWatch alarm on its Errors metric
Account B (secondary, connected per the steps above) — a DynamoDB table with a resource-based policy granting Account A's Lambda role write access

For demo purpose, I deliberatly modify resource policy on dynamodb, from PutItem to GetItem, this introduce lambda errors.

I have Cloudwatch alarm for lambda errors which gets triggers and corresponding devops agent lambda do API call in devops agent space.

In few minutes, devops agent suggest us root cause & remediation

And since I have connectivity with JIRA as well (previous blog), ticket got created in JIRA

I hope this blog gave you how we can do cross-account setup in DevOps agent.

Key Takeaways

Cross-account monitoring exists because most real applications span more than one AWS account, and an agent that can't see the full picture investigates with blind spots.
Access rests on three policies — a trust policy with confused-deputy prevention, the AWS-managed AIDevOpsAgentAccessPolicy, and an Agent-Space-specific inline policy.
Removing a secondary account from the Agent Space doesn't delete its IAM role — budget that into your offboarding checklist.

AWS DevOps Agent Integrations: Slack, Jira, and GitHub

vikasbanage — Tue, 28 Jul 2026 10:07:20 +0000

This is the second post in our AWS DevOps Agent series. In our first blog, we have seen how to set up DevOps Agent & integrated with cloudwatch alarms

An investigation agent is only as useful as the tools it's wired into. In the last post we covered CloudWatch alarms and generic webhooks — the "how it hears about problems" layer. This post covers the "where it talks to your team and your code" layer: Slack for notifications, Jira for ticketing, and GitHub for the code side of things. All three connect differently, and knowing why upfront saves you some head-scratching.

In this blog post, we are going to integrate DevOps agent with Slack, JIRA & Github.

Slack — Built In

Slack is a native integration — no MCP server, no separate app to host. Once connected, AWS DevOps Agent posts key findings, root cause analysis, and generated mitigation plans directly into whichever channels you choose.

Prerequisite: You may need Slack Admin access to install AWS DevOps agent app.

In some cases you may not have admin rights to add this integration, in that scenario, you need to ask Slack admin to install AWS DevOps Agent App for you.

Setup is two steps:

Register the integration — Go to respective agent space, find Slack under Communication and choose Register. You'll be redirected to Slack to authorize the app — install it per workspace.

If no workspace is added, click Register, it will take you to Slack login screen.

Associate channels with your Agent Space — from your Agent Space's Capabilities tab, go to Communications → Slack → Add Slack, and enter the Channel ID for each channel that should receive updates.

In Slack, you can invite AWS DevOps agent or you can also add it under Agent & Apps tab

Now we have everything in place, lets have short demo.

DEMO - CloudWatch Alarm Investigation Notification

In last blog, I have investigation for CloudWatch alarm Cloudfront5xx errors, as soon as investigation started by devops agent I get notification in slack, with timelines, root cause & investigation gaps. No need to login in AWS Console or DevOps agent WebApp

So without switching screens, you get notification in one place.

Jira — Via the Atlassian Rovo MCP Server

Jira isn't a built-in integration, so it connects through Atlassian's own hosted Rovo MCP Server — the same server that also covers Confluence, so connecting Jira gets you Confluence

Step 1 : Configure Rovo MCP in DevOps agent, Agent Space -> Capabilities -> MCP Server

Field	Value
Name	jira-mcp
Endpoint URL	`https://mcp.atlassian.com/v1/mcp`
Enable Dynamic Client Registration	✅ Enabled

Approve access in Jira if you already have admin access for Jira.

In case if you face an issue - you may need to add DevOps Agent Domain in Atlassian Admin under Rovo MCP Server -> Domain. URL : https://*.api.aws/**

Step 2 : Permission for an Agent Space
Once you have MCP access is in place, select which MCP tools you can use. Go for permission which are required only, for demo part I have given access to create tickets for investigations.

Step 3 : Verification

After completing the setup, verify that the integration is working:

Open DevOps agent WebApp
I will be using below prompt : Check for stale/unused resources and create respective Jira ticket under project "cg-helpdesk". In few minutes, it will create Jira tickets

Now to go one step further, let's have Skill in place to create tickets , in DevOps agent WebApp go to Knowledge -> Skills -> Custom Skills. You can create custom skill with below description to have ticket in JIRA, obviously modify it as per your need.

Please note that, its advisable to add Guardrails to avoid data-leakage, ticket noise. As progress happens, you can always modify this skill.

---
name: cloudwatch-alarm-jira-ticket
description: Automatically create Jira tickets in the _my-jira-app_ project when investigating CloudWatch alarms. Use this skill during CloudWatch alarm triage or investigation to log findings, affected resources, severity, and recommended actions as a structured Jira task. Targets the Incident-Triage and Incident-RCA agent types.
---

# CloudWatch Alarm Investigation — Jira Ticket Creator

## When to Use
- A CloudWatch alarm has fired and an investigation is underway
- An investigation has produced findings related to a CloudWatch alarm
- The user asks to "create a Jira ticket", "log this alarm", or "track this in Jira" during an alarm investigation
- Post-investigation wrap-up where findings need to be tracked for remediation

## Prerequisites
- Jira Cloud ID: `1231412535345435345` (demo.atlassian.net)
- Jira Project Key: `HELP` (my-jira-app)
- Issue Type: `Task`
- Use the `jira-mcp_createJiraIssue` tool (available via the jira-mcp MCP server)

## Severity → Priority Mapping

| CloudWatch Alarm Severity | Jira Priority |
|---|---|
| CRITICAL | High |
| HIGH | High |
| MEDIUM | Medium |
| LOW | Low |
| UNKNOWN | Medium |

## 🚧 Guardrails

These rules are mandatory and must be checked before any ticket is created.

### 1. Severity Threshold — Avoid Ticket Noise
**Do not create a Jira ticket for investigations whose alarm severity is below your team's configured threshold.**
- Default threshold: **MEDIUM and above** (i.e. skip LOW severity alarms).
- If the alarm severity is LOW, log a note in the investigation findings stating: "Ticket creation skipped — alarm severity is below the configured threshold (LOW). Adjust the threshold in the skill if this should be tracked."
- Teams should adjust this threshold to match their operational standards.

### 2. One Ticket Per Investigation — No Duplicates
**Never create more than one Jira ticket per investigation.**
- Before creating a ticket, check whether one already exists for this investigation by searching for the investigation ID in the ticket labels or description.
- If a ticket already exists, add a comment to the existing ticket instead of creating a new one.
- If you cannot verify (e.g. search is unavailable), note this in the investigation findings and skip creation rather than risk a duplicate.

### 3. Jira Failure — Surface, Don't Silently Drop
**If the Jira create-issue call fails or Jira is unreachable, do not silently discard the failure.**
- Record the failure explicitly in the investigation notes/findings with:
  - The error message or HTTP status returned
  - The full ticket content that was attempted (summary + description)
  - Instruction: "Jira ticket creation failed — a human must file this ticket manually in cg-helpdesk (HELP)."
- Do not retry more than once automatically.

### 4. No Personal Information in Tickets
**Never include personal details in any Jira ticket field (summary, description, labels, comments).**
- Do not include: personally identifiable information (PII).
- If an alarm or log message contains PII (e.g. a username in a CloudWatch dimension), redact it — use a placeholder like `[REDACTED]` or refer to the resource by its ARN/ID only.
---

GitHub — Native Pipeline Integration

GitHub is a first-class, native integration, and it does more than notify — it lets AWS DevOps Agent correlate deployments with incidents, and (in preview) run release readiness code reviews and automated testing directly against pull requests.

To have integration in place :
Step 1 : Go to respective agent space -> Pipeline -> Add Source -> Register Github

Step 2 : Enter details as per your org or user , you can also directly hit Submit , it will take you to your Github org permission page,

Select Specific repo only :

Step 3 : Configure code review & testing. Here I'm configuring only code review

When Code Review enabled for a repository, DevOps Agent automatically runs a Release readiness code reviews each time a pull request is opened or updated. Review findings appear as inline comments on the pull request. This is enabled by default for all connected repositories.

DEMO

I will change my webapp to orange theme and will create pull request, this triggers automatic code-review in devops agent

From the above screen shot we can see, devops agent able to identify which code change are done and also recommendation part.

Key Takeaways

Slack posts findings and mitigation plans straight into your channels — one workspace install, then channels added per Agent Space.
Jira connects through Atlassian's Rovo MCP Server, add customer skill as per your need to have ticket creation.
GitHub is a native integration that goes beyond notifications — it correlates deployments with incidents and, in preview, runs automated code review and testing on every pull request

I hope this blog gave you a solid idea and a starting point for integrating with all three providers.

Next up in this series: cross-account monitoring, so a single Agent Space can see your whole AWS footprint.

AWS DevOps Agent in Practice: Setup, Investigations, and Integrations

vikasbanage — Tue, 28 Jul 2026 04:50:40 +0000

The worst part of an incident usually isn't fixing it — it's the scramble beforehand, piecing together what's even happening across a dozen tools. That's exactly the gap AWS DevOps Agent is built to close and you already understand the problem AWS DevOps Agent exists to solve. AWS DevOps Agent is AWS's autonomous agent for operations — it investigates incidents the moment they happen, learns your infrastructure well enough to reason about blast radius, and connects into the tools your team already lives in, like Slack and Jira.

In this post we go hands-on: standing one up, running an investigation both on demand and automatically.

Foundation of AWS DevOps Agent

Everything AWS DevOps Agent does happens inside a container called an Agent Space — the boundary that defines exactly which AWS accounts, integrations, and users the agent is allowed to touch. Understanding this boundary is the first thing to get right, because it's the security model the rest of the product builds on.

On top of that boundary sits a deliberately split interface:

For Administrators : To do configuration for agents via console or IaC.
For users : Web App to see investigation, generate reports, create custom agents etc.

Setting Up AWS DevOps Agent

Getting a first Agent Space running comes down to three steps :

Create an Agent Space in the AWS Management Console — Go to AWS Console → AWS DevOps Agent → Create Agent Space. We can also create Devops agent via CLI, IaC - Cloudformation, Terraform.

While creating agent space, you need to mention:
- Name: As per you naming convention
- IAM Role for Agent Space: IAM Role used by agent space access your AWS resources. I will be using default as part of this blog.
- IAM Role for DevOps Agent WebApp: This is IAM role used by WebApp to access Agent Space.
- Encryption Key: AWS managed or CMK. You can mention this only during creation, later cannot be configured.

For a demo or test environment, just use the default IAM role — the AWS-managed AIDevOpsAgentAccessPolicy gives read-only access out of the box, no extra configuration needed. For production, don't carry that forward as-is: watch which additional permissions your test environment's investigations actually reach for, then add only those as scoped inline policies on a custom role — resource-restricted, not account-wide.

Once all field done, hit create.

Accessing DevOps Agent WebApp

On creation of agent, you should able to access it under Access tab :

DevOps agent can integrate with

Communication channel - Slack, PagerDuty, ServiceNow
Telemetry tools - DataDog, NewRelic, Splunk, Grafana, Dynatrace
Source code - Github, Gitlab, AzureDevOps
Add 3rd party MCP servers, Remote agent with A2A support
Webhooks

On-Demand Investigation

You can start an investigation manually any time from the Incident Response tab of your Agent Space web app — either with free-form text describing what you want investigated, or by picking one of three pre-configured starting points:

Latest alarm: investigates your most recently triggered alarm, analyzing underlying metrics and logs to determine root cause
AWS Platform health: Investigate if there are any planned changes from AWS side.

I used prompts : Check for stale/unused resources, it returns nice summary of idle resources

DEMO #1 : Investigate EC2 Alarms

Part of this demo I have EC2 instance created and intentionally ran process to spike CPU (with stress-ng), let's check if DevOps agent able to investigate that.

As we can see above DevOps agent able to detect, it was an intentionally CPU spike. It does it but accessing metrics, cloudtrail logs and co-relating it. Under rootcause tab in incident investigation, you will able to find what are the gaps in investigation , in this example, since we don't have cloduwatch logs for ec2, gap was devops agent was not able detect exact process for CPU spike

Automated Investigation

Rather than waiting for someone to notice and manually start an investigation, AWS DevOps Agent can kick one off itself the moment something fires. So instead of reactive approach, let's go for pro-active approach.

Currently, we cannot target AWS DevOps agent as an action for CloudWatch alarm. So we will do trick here, we will be using lambda as target. Please find below flow we will be configuring.

Webhook Integration - DevOps Agent

To configure generic webhook, follow below steps

Navigate to Agent Space → Capabilities tab → Webhook section → Configure

Generic webhooks: choose Generate webhook, then pick an auth type — HMAC (signs each request, verified via the x-amzn-event-signature header) or API key (sent as Authorization: Bearer ). I will be using HMAC authentication.

Copy the endpoint URL and store the secret/key securely — it's shown only once and can't be retrieved again. We will be configuring this as part of lambda.

Configuring Lambda

Next step is read payload from Cloudwatch alarm and send it to AWS DevOps agent. Below part of lambda code show, forming payload and send payload to devops agent.

I have configured 3 environment variables for tha lambda DEVOPS_AGENT_WEBHOOK_SECRET, DEVOPS_AGENT_WEBHOOK_URL & SERVICE_NAME.
IAM Permission: Use AWS Managed permission AWSLambdaBasicExecutionRole.

Please note that, its not recommended hard code DEVOPS_AGENT_WEBHOOK_SECRET even as environment variable, you should use SSM Parameter Secure String or SecreteManager and configure path of SSM as environment variable.

def _build_payload(event):
    alarm_data = event.get("alarmData", {})
    alarm_name = alarm_data.get("alarmName", "Unknown Alarm")
    alarm_arn = event.get("alarmArn", "")
    account_id = event.get("accountId", "unknown")
    region = event.get("region", "")

    state_block = alarm_data.get("state", {})
    state = state_block.get("value", "ALARM")
    reason = state_block.get("reason", "")
    prev_state = alarm_data.get("previousState", {}).get("value", "")

    config = alarm_data.get("configuration", {})
    desc_text = config.get("description", "")

    namespace, metric_name = _extract_metric(alarm_data)
    dimensions = _extract_dimensions(alarm_data)

    title = f"[{state}] {alarm_name}"
    if region:
        title += f" ({region})"

    desc_parts = []
    if desc_text:
        desc_parts.append(desc_text)
    if reason:
        desc_parts.append(reason)
    description = "\n".join(desc_parts)

    return {
        "eventType": "incident",
        "incidentId": _incident_id(alarm_name, account_id),
        "action": _STATE_TO_ACTION.get(state, "updated"),
        "priority": _STATE_TO_PRIORITY.get(state, "MEDIUM"),
        "title": title,
        "description": description,
        "timestamp": _iso_ts(),
        "service": SERVICE_NAME,
        "data": {
            "source": "aws.cloudwatch",
            "alarmName": alarm_name,
            "alarmArn": alarm_arn,
            "state": state,
            "previousState": prev_state,
            "reason": reason,
            "accountId": account_id,
            "region": region,
            "namespace": namespace,
            "metricName": metric_name,
            "dimensions": dimensions,
        },
    }

def _post_webhook(payload, request_id):
    timestamp = _iso_ts()
    payload_str = json.dumps(payload, separators=(",", ":"), ensure_ascii=False)
    signature = _sign(WEBHOOK_SECRET, timestamp, payload_str)

    logger.info("[%s] Sending to DevOps Agent. preview=%s",
        request_id, payload_str[:1200])

    req = urllib.request.Request(
        url=WEBHOOK_URL,
        data=payload_str.encode("utf-8"),
        method="POST",
        headers={
            "Content-Type": "application/json",
            "x-amzn-event-timestamp": timestamp,
            "x-amzn-event-signature": signature,
        },
    )
    with urllib.request.urlopen(req, timeout=20) as resp:
        body = resp.read().decode("utf-8", errors="replace")
        logger.info("[%s] Webhook response status=%s body=%s",
            request_id, resp.status, body[:1000])
        return {"status": resp.status, "body": body[:1000]}

Configure CloudWatch Alarm Action

Once AWS lambda is in place, you can edit existing CloduWatch alarm with action.

DEMO 2: Automated Investigation

I have Webapp- URL-shortner behind CloudFront. Part of this webapp, lambda process requests, to intentional failure in this app, I have set lambda concurrency to 0 which disable lambda execution and clodufront throws 5xx errors.

I have created CloudFront5xx alarm which gets triggered when website throws 5xx errors.

From below screenshot, we see when alarm gets triggered respective lambda which sent payload to devops agent

On checking further in AWS DevOps Agent Space, we can see agent started with investigation. Open DevOps agent WebApp, under Incident you should able to see investigation has started.

We can also see how devops agent approached investigation in timeline e.g.

In few minutes, DevOps agent came up with root cause of this issue.

From the above screenshot we able to see accuracy of DevOps agent is ~100%.

For most of the scenarios I tried, accuracy for DevOps agent is always > 90% and this is quite impressive.

Key Takeaways

Everything runs inside an Agent Space, isolated across account access, user access, data, and chat history — that's the foundation everything else builds on.
Setting up AWS DevOps Agent
You can investigate on demand from the web app, or let the agent trigger automatically from Cloudwatch alarm via Webhooks

Hopefully this gives you a solid picture of how to configure AWS DevOps Agent for CloudWatch alarms.

Next up in this series: integration with Slack, Jira, and GitHub.

AWS Resource Control Policies (RCPs) Explained: A Practical Guide to Resource-Level Security

vikasbanage — Wed, 14 Jan 2026 08:54:12 +0000

Modern AWS environments are built for scale — multiple accounts, shared teams, and resources that need to operate across boundaries. While this flexibility enables agility, it also introduces a critical governance challenge: even when IAM policies and SCPs are correctly configured, resource policies can still be misconfigured, resulting in overly broad access or unintended exposure.

That gap is exactly what Resource Control Policies (RCPs) were created to address.

RCPs let you enforce non-negotiable rules directly on resources, such as:

This resource can never be shared outside the organization
Only approved identities can assume roles here
Even admins can’t break these rules

They don’t replace IAM or SCPs — they fill the last missing governance layer.

Layer	What it controls
IAM	Which principals can perform which actions on which resources, within organizational guardrails.
SCP	The maximum permissions IAM principals in an account or OU can ever have; they restrict but do not grant access.
RCP	The maximum permissions that can ever apply to specific resources in accounts or OUs, regardless of which principal calls them.

Let's now jump to hands-on for RCP, how it works.

Note : It is highly advisable to test RCP on dev/test account before applying. Don't apply directly at Root level.

Hands-On

In this demo, we will be covering two scenarios:

Enforce HTTPS-only across all S3 buckets
Prevent external accounts from decrypting KMS keys. or Disable cross-account use of KMS keys.

Scenario #1 : Enforce HTTPS-only across all S3 buckets

I have created S3 bucket and intentionally set over permissive access as follows. I have uploaded simple file which has text RCP HTTPS test


    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "S3rcppolicy",
            "Effect": "Allow",
            "Principal": "*",
            "Action": [
                "s3:GetObject",
                "s3:GetObjectVersion",
                "s3:PutObject"
            ],
            "Resource": "arn:aws:s3:::XXXX-XXXXX-demo-XXXXXXX/*"
        }
    ]
}

Now if you access files using HTTP and HTTPs, both will work :

HTTP Test:

HTTPS test

Now, let's apply RCP at account level to make sure only HTTPs access is allowed

After attaching RCP, you can see below HTTP requests gets AccessDenied (highlighted part) but HTTPS works.

As you have also observed that, once we apply RCP its applicable to current and new resource policies as well. So while applying RCP make sure that which resources are intentionally shared across org.

Scenario #2 : Prevent external accounts from decrypting KMS keys.

This example blocks the decryption of AWS KMS (Key Management Service) keys for any principal except whitelisted principal in policy.

Part of this demo I have create KSM key and given access for decrypt to other account in KMS policy.

{
      "Sid": "Enable IAM User Permissions",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::714XXXXXXXXX:root"
      },
      "Action": "kms:*",
      "Resource": "*"
    },
    {
      "Sid": "Allow use of the key",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::599XXXXXXXXX:root"
      },
      "Action": [
        "kms:Encrypt",
        "kms:Decrypt",
        "kms:ReEncrypt*",
        "kms:GenerateDataKey*",
        "kms:DescribeKey"
      ],
      "Resource": "*"
    },

I'm encrypting one file using KMS key in AccountA(714XXXXXXXXX) and will try to decrypt it using assuming IAM role from AccountA(714XXXXXXXXX) and AccountB(599XXXXXXXXX).

Without RCP applied, we able to decrypt it.

AccountA(714XXXXXXXXX)

AccountB(599XXXXXXXXX)

Next, we apply the RCP directly to Account A (714XXXXXXXXX). With this policy in place, access is restricted so that only users or roles from Account A can perform the allowed actions, regardless of how individual resource or IAM policies are configured.

{
  "Effect": "Deny",
  "Principal": "*",
  "Action": [
    "kms:Decrypt",
    "kms:Encrypt",
    "kms:GenerateDataKey"
  ],
  "Resource": "*",
  "Condition": {
    "StringNotEquals": {
      "aws:PrincipalAccount": [
        "714XXXXXXXXX"
      ]
    }
  }
}

If I perform same operation for decryption, I get error for AccountB(599XXXXXXXXX) with explicit Deny in RCP.

Before implementing Resource Control Policies - RCPs

RCPs are useful feature from AWS to design data perimeter in AWS, it protect data, block breaches, and enforce real cloud perimeters — but if deployed carelessly, they can also break production workloads. We should must create and test RCPs before applying to critical workloads.

Before you deploy any RCP, make sure it passes through a formal test cycle:

What to test

Use AWS IAM Access Analyzer to understand:

Which resources are currently public
Which are shared externally
Which identities depend on cross-account access

This helps you avoid breaking legitimate access paths.

Then validate policy logic using:

IAM Policy Simulator
Controlled sandbox accounts
Dev OUs before prod

Also test interactions between SCPs and RCPs.

Remember: a deny from either one will block access.

Other High-Impact RCP Use Cases to Explore

AWS maintains a fantastic repository of real-world RCP patterns.

🔐 Enforce Org-Only STS Access
🔑 Lock Down OIDC Providers (GitHub Actions, etc.)
🗄️ Block External Sharing of Data Stores

When used thoughtfully, RCPs close the last remaining gaps around data access, cross-account trust, and misconfigurations.

Good security isn’t about trusting people — it’s about designing systems that remain safe even when mistakes happen.

Thanks for reading — hope this helps you build safer, more resilient AWS environments.

CloudWatch Investigations: Your AI-Powered Troubleshooting Sidekick

vikasbanage — Sun, 04 Jan 2026 20:04:30 +0000

Remember those 3 AM incidents when you’re frantically switching between dashboards, digging through logs, and wondering if you should just restart everything? We all have been through the situation where we worked in non-business hours, weekends, midnights to troubleshoot production issues and its quite energy draining task. What if in this GenAI world we get AI assistant that works 24*7 and guide us through the chaos.

Enter CloudWatch Investigations – a generative AI-powered feature that’s changing how we handle incidents in AWS environments.

When something breaks, instead of you jumping between CloudWatch metrics, logs, deployment history, CloudTrail, X-Ray, and health dashboards, CloudWatch Investigations does the first round of detective work for you.

It uses generative AI to scan your system’s telemetry and quickly surface:

the metrics that look suspicious
the logs that matter
recent deployments or config changes
and even possible root-cause hypotheses, especially when multiple resources are involved

All of this is presented visually, so you can see how things are connected instead of guessing.

It's like having an extra team member who's been staring at your system architecture 24/7 .

Let's get started to check how to configure CloudWatch investigation.

Getting Started

In AWS Console, Go to CloudWatch → AI Operation (left pane), if you are configuring account for first time, you need to do setup
- Creating Investigation Group
  - Configure Retention days: For how much time you would like keep investigation. Please note that The retention cannot be changed once it is configured.
  - Customise Encryption: You can have customer managed key for encryption. But make sure you give permission to access key.
  - Next, it will create new role with required permission for investigation. These will be read-only permission which needed for investigation. You can also create new role here. By default, it uses : AIOpsAssistantPolicy, AmazonRDSPerformanceInsightsFullAccess and AIOpsAssistantIncidentReportPolicy**

Once the investigation group is created, you will be able to see Optional Enhanced configuration

Under Enhanced integration, you will be able to include tags related to application. This will help Cloudwatch to narrow down investigation. This is quite useful setting as its efficient to narrow down investigation.
Access to CloudTrail event, to help CloudWatch investigations better discover relevant change events.
Optionally, X-ray, Application Signals and EKS access entries.

DEMO

Now, we have configuration ready, let's start with demo. Part of this blog, I have simple Event booking app as follows.

User books appointment by providing details and selecting available slots, admin approves/rejects requests.

Disruption in Application

Part of this demo I have modified Lambda role permission where I have removed KMS permission.

Now imagine scenario, suddenly users started reporting errors they are not able see the slots, it throws an errors. Also Admin not able to see any appointments at all.

As we know application design, entry point for application is CloudFront, we start checking Cloudfront and see there is increase in 5xx errors.

Starting Investigation

Under CloudWatch metrics 5xx, you can start already start investigation why its throwing 5xx errors.

It will pick the timestamp automatically or you can adjust from what time you would like to start investigation.

Once investigation is started, it will take 10-15 minutes to finish investigation, we can also view progress of investigation. But instead, we can always start communicating user/business or start other parallel activities.
On completing investigation, it correctly pointed out what went wrong and why we are getting 5xx errors 🥳🥳🥳

As we can use under Root Cause Summary, it was IAM configuration issue which was causing issue.

ANALYSIS: This failure pattern represents an IAM configuration issue rather than a service degradation, as evidenced by the specific KMS permission errors and the NEW occurrence pattern indicating a recent permission change affecting the eventap staging service components.

We have root cause in 15 minutes. This is huge advantage for anyone who works on production system and need to keep system running.

Going one step further, instead of checking metrics every time and start investigating. We can have CloudWatch Alarm in place, as soon as resources metrics gets ALARM start, CloudWatch Investigation will get automatically started and I did same.

I hope this blog gives you a good idea of how you can get started with CloudWatch Investigations. There may be moments where you don’t fully agree with the AI’s suggestions — and that’s perfectly fine. You’re always in control. You can accept what makes sense, discard what doesn’t, and guide the investigation in the right direction.

The beauty is that you can start small with zero setup, and then gradually level up by adding richer telemetry, cross-account visibility, and automation runbooks. Over time, this leads to fewer guesswork-driven fixes, faster MTTR, and much calmer incident calls — even at 3 AM.

Instead of panic-driven troubleshooting and endless tab-hopping across metrics, logs, and dashboards, you get context first: what changed, what’s related, and what’s most likely broken.

Thanks for reading, and happy troubleshooting 🚀

How to Use Amazon SNS Data Protection Policies to Prevent Sensitive Data Leakage

vikasbanage — Sat, 29 Nov 2025 19:05:06 +0000

When we build things using event-driven architecture, we almost always run into Amazon SNS and for good reason. It’s simple, scalable, and makes it incredibly easy to fan out messages to multiple subscribers.

Imagine a fintech or healthcare application that sends transaction alerts or patient updates via SMS or email. These messages may accidentally include sensitive information such as account details, names, or dates of birth. Encrypting SNS topics and applying strict access controls helps ensure compliance, but it’s equally important to prevent sensitive data from leaking into messages themselves.
That’s where ** SNS Message Data Protection** comes in — because your architecture isn’t just about fast delivery, it’s also about secure delivery.

In this blog, I will walk through how we can protect personal and sensitive information while sending notifications through SNS using Data Protection Policies.

Data Protection in SNS

Amazon SNS uses data protection policies to identify and manage sensitive data (like PII and PHI) in message payloads with Predefined or custom data identifiers by using Machine learning and pattern matching.

Each policy allows you to define operations based on detection:

Audit – Log findings without interrupting delivery
De-identify – Mask or redact sensitive data
Deny – Block messages containing sensitiv e data

A policy is defined in JSON format and includes elements like:

DataDirection (Inbound/Outbound)
Principal (IAM identity publishing/subscribing)
DataIdentifier (e.g., name, phone number)
Operation (Audit, De-identify, Deny)

Only one data protection policy per SNS topic is allowed, but it can have multiple statements. This helps organizations enforce privacy controls and reduce compliance risks.

Why should I use message data protection?

Introducing SNS Data Protection into your governance, risk, and compliance programs helps you automatically detect, prevent, and control data leakage. It safeguards regulated data (PII/PHI) and reduces the overhead of building your own detection or masking pipeline.

Defining SNS topics with policy

We define three SNS topics — each configured with its own data protection policy: Audit, De-identify, and Deny. These sample policies demonstrate how SNS handles sensitive data under different rules.

Below are the three sample data protection policies used in this blog—Deny to block sensitive data, Audit to log sensitive content without stopping delivery, and De-identify to automatically mask regulated fields before the message is published.

Audit - Data protection policy

This policy detects email, date of birth, and credit card numbers.
If found, the finding is logged in CloudWatch but delivery continues.

One important thing to note, CloudWatch log group name in this case must have prefix /aws/vendedlogs/

{
  "Description": "Audit sensitive data without blocking delivery",
  "Version": "2021-06-01",
  "Statement": [
    {
      "DataDirection": "Inbound",
      "DataIdentifier": [
        "arn:aws:dataprotection::aws:data-identifier/EmailAddress",
        "arn:aws:dataprotection::aws:data-identifier/DateOfBirth",
        "arn:aws:dataprotection::aws:data-identifier/CreditCardNumber"
      ],
      "Operation": {
        "Audit": {
          "FindingsDestination": {
            "CloudWatchLogs": {
              "LogGroup": "/aws/vendedlogs/sns-audit/"
            }
          },
          "SampleRate": "99"
        }
      },
      "Principal": [
        "*"
      ],
      "Sid": "AuditSensitiveData"
    }
  ],
  "Name": "sns-audit-policy"
}

De-Identify : Data protection policy

This policy masks sensitive fields using # characters. Subscribers never see actual sensitive data.

{
  "Description": "Mask or redact sensitive data",
  "Version": "2021-06-01",
  "Statement": [
    {
      "DataDirection": "Inbound",
      "DataIdentifier": [
        "arn:aws:dataprotection::aws:data-identifier/EmailAddress",
        "arn:aws:dataprotection::aws:data-identifier/DateOfBirth",
        "arn:aws:dataprotection::aws:data-identifier/CreditCardNumber"
      ],
      "Operation": {
        "Deidentify": {
          "MaskConfig": {
            "MaskWithCharacter": "#"
          }
        }
      },
      "Principal": [
        "*"
      ],
      "Sid": "DeidentifySensitiveData"
    }
  ],
  "Name": "sns-deidentify-policy"
}

Deny : Data protection policy

This policy blocks the publish request entirely if sensitive data is detected.

{
  "Description": "Block messages containing sensitive data",
  "Version": "2021-06-01",
  "Statement": [
    {
      "DataDirection": "Inbound",
      "DataIdentifier": [
        "arn:aws:dataprotection::aws:data-identifier/EmailAddress",
        "arn:aws:dataprotection::aws:data-identifier/DateOfBirth",
        "arn:aws:dataprotection::aws:data-identifier/CreditCardNumber"
      ],
      "Operation": {
        "Deny": {}
      },
      "Principal": [
        "*"
      ],
      "Sid": "DenySensitiveData"
    }
  ],
  "Name": "sns-deny-policy"
}

Demo

I have created simple lambda function to test these topics:

import boto3
import os
import json
import logging

sns = boto3.client('sns')
logger = logging.getLogger()
logger.setLevel(logging.INFO)

def lambda_handler(event, context):
    message = {
        "patientId": "PAT123456",
        "name": "John Doe",
        "dob": "12-01-2012",
        "diagnosis": "Flu"
    }

    topics = ["AUDIT_TOPIC_ARN", "DEIDENTIFY_TOPIC_ARN", "DENY_TOPIC_ARN"]
    results = {}

    for topic_env in topics:
        topic_arn = os.environ.get(topic_env)

        try:
            response = sns.publish(
                TopicArn=topic_arn,
                Message=json.dumps(message)
            )
            results[topic_env] = {
                "status": "success",
                "messageId": response.get("MessageId")
            }
            logger.info(f"Published to {topic_env}: {response.get('MessageId')}")

        except sns.exceptions.InvalidParameterException as e:
            # Common case: DENY_TOPIC_ARN rejects sensitive fields
            logger.error(f"[{topic_env}] Sensitive data detected or invalid parameter: {str(e)}")
            results[topic_env] = {"status": "failed", "error": "Sensitive data not allowed"}

        except sns.exceptions.AuthorizationErrorException as e:
            logger.error(f"[{topic_env}] Access denied: {str(e)}")
            results[topic_env] = {"status": "failed", "error": "Access denied"}

        except Exception as e:
            # Catch any unexpected exception
            logger.error(f"[{topic_env}] Unexpected error: {str(e)}")
            results[topic_env] = {"status": "failed", "error": str(e)}

    return {
        "status": "completed",
        "results": results
    }

I'm sending below JSON where I have date of birth as personal information.

  {
        "patientId": "PAT123456",
        "name": "John Doe",
        "dob": "12-01-2012",
        "diagnosis": "Flu"
    }

On testing I get below results ,

For Audit, I do get message but its get logged under CloudWatch log group

For De-identify, we get masked message for date of birth

And for Deny, we get Access Denied error.

**_> Access denied: An error occurred (AuthorizationError) when calling the Publish operation: One or more data identifiers were found_**

Below is summary for three data protection policy types:

Policy Type	Purpose	What Happens When Sensitive Data Is Detected?	Impact on Message Delivery	Ideal Use Case
Audit	Monitor sensitive data exposure	Logs findings to CloudWatch using `/aws/vendedlogs/`	✔️ Delivered	Compliance monitoring, security insights, debugging sensitive data flow
De-Identify	Mask or redact sensitive data before delivery	Sensitive fields are replaced with `#` or a chosen mask	✔️ Delivered (masked)	Sending events to analytics systems, external subscribers, or downstream apps that shouldn't see PII
Deny	Prevent sensitive data from being published	Publish request is blocked with an `AuthorizationError`	❌ Not delivered	Strict compliance environments (PCI/HIPAA), preventing accidental PII leakage

I hope this walkthrough gives you a clear understanding of how SNS Data Protection policies work and where they can be applied across real-world scenarios. By using these capabilities, teams can significantly reduce compliance risks, strengthen data governance, and build more secure, trustworthy systems—without sacrificing the speed or scalability of their event-driven architecture.

Stay secure. Stay responsible. Keep building.

How to Automate IAM Best Practices in CI/CD with IAM Access Analyzer

vikasbanage — Sat, 19 Apr 2025 05:15:09 +0000

Managing IAM (Identity and Access Management) policies securely is one of the most important parts of working with AWS. Developers may accidentally create overly-permissive policies that grant more access than necessary — for example, allowing iam:PassRole to all roles, or opening up sts:AssumeRole without restriction. Without proper checks in place, these risky permissions can silently make their way into your production environment.

In a organizations with multiple accounts, the impact of such mistakes can multiply. That’s why having strong guardrails like IAM Access Analyzer become critical — ensuring that only safe and intentional access is allowed.

🛡️ Before we go further, it's important to understand that AWS Service Control Policies (SCPs) and Resource Control Policies (RCPs) are the first line of defence in any AWS multi-account setup. They define the maximum permissions any user or role can have, regardless of their IAM policies. This blog will not cover SCPs and RCPs, we’ll focus on validating IAM policies during development to ensure we follow least privilege and catch access issues early.

In this blog, we’ll explore how to bring IAM policy validation into CI/CD pipeline using AWS IAM Access Analyzer. This is an example of shift-left security — catching misconfigurations as early as possible, before anything gets merged or deployed.

We'll walk through:

Setting up a GitHub Actions workflow that validates IAM policies.
Using IAM Access Analyzer checks.
Building CloudFormation templates that intentionally pass and fail to understand how policy validation works in practice.

IAM Access Analyzer

IAM Access Analyzer is a security feature in AWS that helps ensure your IAM policies follow best practices and don’t grant unintended access. It supports several capabilities like:

Validating IAM policies against AWS best practices or security standards.
Identifying unused permissions
Analyzing external sharing of AWS resources
Detecting public access to resources.
Generating policies based on CloudTrail access logs

In this blog, we’ll focus specifically on Custom policy checks, which help to validate IAM policies against your organizations security standards — like least privilege or restricted actions. IAM Access Analyzer provides custom policy check APIs to validate IAM policies:

- CheckNoNewAccess – Detects if a policy introduces new permissions compared to a baseline.
- CheckAccessNotGranted – Ensures specific actions or access are not allowed.
- CheckNoPublicAccess – Identifies if a resource policy allows public access.

👉 IAM Access Analyzer custom policy check is a paid feature. As of now, AWS charges $0.0020 per API call for these checks. For example, you make 10,000 calls each month to the IAM Access Analyzer APIs to run custom policy checks across 5 accounts signed up for consolidated billing with AWS Organizations, cost will be $0.0020*10,000 API calls = $20 per month

Now let's start with playing around IAM Access Analyzer. Github code for this blog.

Pre-Requisite

AWS Account
Github Account

CI-CD Flow

Developer pushes changes to repo.
Github action performs IAM Access Analyzer validation.
If policy created by developer violates, workflow gets failed.
If policy created by developer is as per security standard, workflow proceeds with new checks and depployment.

Hands-On

Step 1 - Configure AWS and Github

In order to, configure Github action, we need to first integrate AWS Account and Github, basically we need to establish trust. To do that follow below steps:

If you already have github configured with AWS, you can skip this step.

Login into AWS account, go to IAM → Identity providers →Add provider.
- For the provider URL: Use https://token.actions.githubusercontent.com
- For the "Audience": Use sts.amazonaws.com
Once done, create a role for GitHub. We need to set trust policy for role as below. You need to replace values as per your accounts.
- Principal should be ARN of Identity provider you configured above.
- For Condition, paste github repo link.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::012345678910:oidc-provider/token.actions.githubusercontent.com"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "token.actions.githubusercontent.com:aud": "sts.amazonaws.com",
          "token.actions.githubusercontent.com:sub": "repo:<githuborg>/<github-repo>:*"
        }
      }
    }
  ]
}

For this role, I have assigned below permission :

AmazonS3ReadOnlyAccess
IAMAccessAnalyzerReadOnlyAccess

Depending on your scenario, you can add more permission like creating IAM roles, policies.

Step 2 - Configure Github Secrets

While running Github action, we will be using role which get assumed to perform necessary actions. But instead of directly hardcoding role ARN we will be configuring it as secrete. For that

Go to your repo → Setting → Secrete & Variable → New repository secretes. Enter name for secrets and paste role ARN

We will be creating two secretes:

IAM Role : This will be used by Github action.
Reference policy Object S3 URL : This will be reference policy which we will be using for validation. As good practice, I will be storing this reference policy on restricted S3 bucket. Part of workflow, reference policy gets downloaded. You can create dedicated S3 bucket for this or use existing one, make sure github role have permission to access it.

Step 3 - Reference Policy

I will be using below reference policy. This reference policy allows all actions by default but explicitly denies a list of CloudFormation actions (like CreateStack, UpdateStack, DeleteStack, etc.) on stacks with names under platformteam/*. It’s designed to protect critical infrastructure owned by the platform team from unauthorized changes, even if general CloudFormation access is permitted elsewhere.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "*",
            "Resource": "*"
        },
        {
            "Effect": "Deny",
            "Action": [
                "cloudformation:CancelUpdateStack",
                "cloudformation:ContinueUpdateRollback",
                "cloudformation:CreateChangeSet",
                "cloudformation:CreateStack",
                "cloudformation:DeleteChangeSet",
                "cloudformation:DeleteStack",
                "cloudformation:DescribeChangeSet",
                "cloudformation:DescribeChangeSetHooks",
                "cloudformation:DescribeStackEvents",
                "cloudformation:DescribeStackResource",
                "cloudformation:DescribeStackResourceDrifts",
                "cloudformation:DescribeStackResources",
                "cloudformation:DescribeStacks",
                "cloudformation:DetectStackDrift",
                "cloudformation:DetectStackResourceDrift",
                "cloudformation:ExecuteChangeSet",
                "cloudformation:GetStackPolicy",
                "cloudformation:GetTemplate",
                "cloudformation:GetTemplateSummary",
                "cloudformation:ListChangeSets",
                "cloudformation:ListStackResources",
                "cloudformation:RecordHandlerProgress",
                "cloudformation:RollbackStack",
                "cloudformation:SetStackPolicy",
                "cloudformation:SignalResource",
                "cloudformation:TagResource",
                "cloudformation:UntagResource",
                "cloudformation:UpdateStack",
                "cloudformation:UpdateTerminationProtection"
            ],
            "Resource": "arn:aws:cloudformation:*:*:stack/platformteam/*"
        }
    ]
}

Step 4 - Github Workflow

With below Github workflow:

Runs on pull requests and pushes to the main branch
Assumes AWS IAM role using OIDC authentication
Downloads reference policies from S3 using GitHub secrets
Performs two IAM Access Analyzer checks:
- ✅ CloudFormation Access Check – Ensures no new permissions are introduced (CHECK_NO_NEW_ACCESS)
- ✅ Principal Check – Verifies only approved principals can assume the role (CHECK_NO_NEW_ACCESS on trust policy)

name: cfn-policy-validator-workflow
on:
  pull_request:
    types: [opened, review_requested]

  push:
    branches:
      - 'main'

permissions:
  id-token: write
  contents: read
  issues: write

jobs: 
  cfn-iam-policy-validation: 
    name: iam-policy-validation
    runs-on: ubuntu-latest
    permissions: write-all
    steps:
      - name: Checkout code
        id: checkOut
        uses: actions/checkout@v4

      - name: Configure AWS Credentials
        id: configureCreds
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: ${{ secrets.OIDC_IAM_ROLE }}
          aws-region: us-east-1
          role-session-name: GitHubSessionName

      - name: Fetch reference policy from s3
        id: getReferencePolicy
        run: |
          aws s3 cp ${{ secrets.REFERENCE_IDENTITY_POLICY_CLOUDFORMATION }} ./cloudformation-policy-reference.json
          aws s3 cp ${{ secrets.REFERENCE_IDENTITY_POLICY_PRINCIPAL }} ./allowlist-account-principal.json
        shell: bash

      - name: CloudFormation-Access-Checks
        id: run-aws-check-no-new-access
        uses: aws-actions/cloudformation-aws-iam-policy-validator@v1.0.1
        with:
          policy-check-type: 'CHECK_NO_NEW_ACCESS'
          template-path: './cloudformation-sample/cfn-iam-pass-role-sample.yaml'
          reference-policy: './cloudformation-policy-reference.json'
          reference-policy-type: "IDENTITY"
          region: us-east-1
      - name: New-Principal-Checks
        id: run-aws-check-no-new-access-assumerole
        uses: aws-actions/cloudformation-aws-iam-policy-validator@v1.0.1
        with:
          policy-check-type: 'CHECK_NO_NEW_ACCESS'
          template-path: './cloudformation-sample/cfn-iam-role-principal-sample.yaml'
          reference-policy: './allowlist-account-principal.json'
          reference-policy-type: "RESOURCE"
          region: us-east-1

CloudFormation Template - Failed

Now below template, grants CloudFormation permissions to all stacks (Resource: ''), violating the reference policy by potentially allowing changes to `platformteam/` stacks.

# lambda-pass-template.yaml
AWSTemplateFormatVersion: '2010-09-09'
Resources:
  LambdaFriendlyRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: LambdaSafeCFRole
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service: lambda.amazonaws.com
            Action: sts:AssumeRole
      Policies:
        - PolicyName: CFNonPlatformTeamAccess
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action:
                  - cloudformation:CreateStack
                  - cloudformation:UpdateStack
                  - cloudformation:DeleteStack
                Resource: '*'

After pushing changes to main (ideally pushing to main is not best practice but for this blog only i'm doing), we will see github workflow gets failed as policy violates our reference. policy.
When a custom policy check fails, IAM Access Analyzer returns the statement ID (Sid) and statement index of the specific policy statement that caused the failure. The index is zero-based (starting from 0).

CloudFormation Template - Pass

With below template, a lambda function CloudFormation permissions scoped to devteam/* stacks, staying compliant with the reference policy by avoiding access to protected platformteam/* stacks.

# lambda-pass-template.yaml
AWSTemplateFormatVersion: '2010-09-09'
Resources:
  LambdaFriendlyRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: LambdaSafeCFRole
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service: lambda.amazonaws.com
            Action: sts:AssumeRole
      Policies:
        - PolicyName: CFNonPlatformTeamAccess
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action:
                  - cloudformation:CreateStack
                  - cloudformation:UpdateStack
                  - cloudformation:DeleteStack
                Resource: arn:aws:cloudformation:us-east-1:*:stack/devteam/*

I hope this example gave you an idea about implementation and how it works. If you would like play more on this, you can find reference repo.

Validating IAM policies early in the development process helps prevent security risks before they reach production. By integrating IAM Access Analyzer with GitHub Actions, you can enforce least privilege, block overly-permissive changes, and shift security left — all without slowing down development.

Thanks, cloud builder! Now go forth and validate those IAM policies like a pro. 🔐🛠️

How to monitor Unused Amazon EBS Volumes

vikasbanage — Thu, 30 Jan 2025 05:05:17 +0000

EBS storage is a fundamental component of cloud storage, often used as the primary storage attached to EC2 instances. Even though an EBS volume is attached to an instance, it has a separate lifecycle. If we don’t monitor EBS storage for unused volumes, cloud costs can escalate significantly.

Unless the Delete on Termination option is selected during instance creation, terminating an EC2 instance detaches the EBS volume but doesn’t delete it. Specially in development and testing environments, where EC2 instances are frequently launched and terminated, this often results in a large number of unused EBS volumes.

These unused EBS volumes continue to accrue charges in your AWS account, regardless of whether they are being used.

So why Delete Unused EBS Volumes:

EBS volumes incur charges based on storage usage. Removing unused volumes reduces unnecessary costs.
Regularly cleaning up unused resources simplifies account management and reduces clutter.
Unused volumes may contain sensitive or outdated data. Deleting these volumes can prevents accidental exposure.

In this blog, we will explore how to configure an AWS Config Rule and set up an automatic remediation action using AWS Systems Manager Automation to delete unused EBS volumes. Main goal of this blog is to bring awareness about unused EBS volumes in AWS environment. Deleting EBS volume can be optional step.

Note: It is not advisable to use this solution directly in production. It is always good to test solution and also decide what data strategy within team/business like retention period etc.

Solution Overview

Detects unused EBS volumes using the AWS-managed Config Rule.
Automatically creates a snapshot of the volume. I would recommend this step if you are going to delete unused volumes.
Deletes the unused volume.

Implementation Steps

You can refer this Github repo and deploy required resources using cloudformation.

After cloning this repo, you can deploy Cloudformation template via console. It takes two parameter:

Config rule name - String
IsSnapshotRequired - Boolean.

This Cloudformation will deploy:

IAM Role which will be assumed during detection and remediation.
AWS Config Rule with remediation action.

Remediation is AWS Managed automation document so we don't build our own as of now.

One you deployed cloudormation template, confirm config rule creation by going AWS Config console. AWS Config - Rules

Test the Solution

To test the solution, I'm creating an EC2 instance with 2 EBS volumes attached. While creating an EC2 instance, make sure for EBS volume you select Delete on Termination value No. To see this option, you need to click Advance under Storage(volume) option while creation an EC2 instance.

After creating an EC2 instance, in few minute AWS Config rule get evaluated automatically. You should able to see volumes are in Compliant state as they are attached to EC2 instance.

Now delete EC2 instance. Once EC2 instance deleted, EBS volume status changed to Available. Since volumes are not attached to any EC2 instance now, config rule make these volumes Noncompliant and start deleting it by taking snapshot.

You can check status of remediation by going to System Manager -> Automation

Below you can also EBS snapshot has also been created.

Conclusion

I hope this blog gives you an idea how we can leverage AWS Config and AWS Systems Manager to manage not attached EBS volumes. If you don't want to delete volume, that can also done. You can just make volumes Noncompliant, just fetch list and send it to respective teams. Feel free to modify solution as per your need.

Stay secure, optimised in Cloud!!!

How to Set Up Cross-Account EventBridge

vikasbanage — Mon, 06 Jan 2025 15:43:27 +0000

Introduction

Amazon EventBridge is a powerful event bus service that makes it easier to build event-driven architectures. It allows you to connect different AWS services or even external SaaS applications through a simple and scalable setup.

While EventBridge is incredibly versatile, its ability to target endpoints or consumers is typically restricted to the same AWS account.
The exception is an event bus in a different account, which can be a valid target.

To achieve this, events must be pushed from the source account's event bus to the destination account's event bus. This cross-account communication is essential for managing critical events and ensuring centralized visibility and control.

In this blog, we'll demonstrate this with an example: when a security service in AWS is disabled, it generates an event that is forwarded to a central monitoring account event bus, triggering an alarm. This approach helps maintain a robust, scalable, and secure event-driven architecture across AWS accounts.

Architecture and Workflow

In given architecture :

We have two accounts workload account (source account for event) and central-monitoring account (destination account).
If Security service like SecurityHub disable in workload account, event rule on default event bus will get triggered.
This event rule has target and respective permission to send event to central-monitoring account event custom bus. You can configure default event bus as well.
Custom event bus in central-monitoring account have similar event rule as above. This event rule triggers Lambda function , that can enrich event and send it to SNS.
On SNS, if you subscription conifgured, you will get notification.

Deployment

To create resources via Cloudformation you can refer this Github repo.
Part of this stack below resources get created

Central Account - Receiver

Custom event bus with permission for workload i.e. source account.

{
  "Version": "2012-10-17",
  "Statement": [{
    "Sid": "AllowSourceAccountPutEvents",
    "Effect": "Allow",
    "Principal": {
      "AWS": "arn:aws:iam::99999999999:root"
    },
    "Action": "events:PutEvents",
    "Resource": "arn:aws:events:us-east-1:111111111:event-bus/CentralMonitoringBus"
  }]
}

Note: Here in principle I have specified root of workload account. It means any entity from workload account can publish message. My advice would be to have specific role.

Event rule that matches event related to security hub

{
  "Version": "2012-10-17",
  "Statement": [{
    "Sid": "AllowSourceAccountPutEvents",
    "Effect": "Allow",
    "Principal": {
      "AWS": "arn:aws:iam::714258651552:root"
    },
    "Action": "events:PutEvents",
    "Resource": "arn:aws:events:us-east-1:11111111:event-bus/CentralMonitoringBus"
  }]
}

Lambda function that will be the target of above rule. This lambda will extract details from event and send it to SNS.

Workload Account - Source

Event rule on default event bus. It get trigger when security hub is disable. It has target set to custom event bus of central account.

Demo

In workload account, I disabled security hub, this triggered event rule in workload account.

This event sent further details to central account where event rule triggers lambda function that extract details from event like Accountid, service which is getting disabled. These details will get send to SNS. For this demo I have created Email subscription under SNS topic. So I get notification over an email. As per your need you can set communication channel.

Conclusion

I hope this blog gives you an idea how we can set cross account communication between AWS accounts via even bridge. Event-bridge is one of the important service in AWS which we can use effectively to build scalable application.

Guide to AWS Certifications: Choosing the Right Path for Your Role

vikasbanage — Sun, 29 Dec 2024 17:13:31 +0000

As cloud computing continues to grow, AWS certification is one of the important step for IT/Non-IT professionals who want to advance their careers. AWS provides a range of certifications designed for various roles.

In this blog, we’ll explore the AWS certification paths for several key positions, helping you decide which certifications to pursue based on your career goals.

If you are from Non-IT background, working in sales and marketing then it is always good to Start with AWS Cloud Practitioner Certification. AWS Certified Cloud Practitioner that validates foundational knowledge of AWS Cloud and terminology.
If you are from IT background, having 1–3 years of IT/STEM experience. I would suggest to skip AWS Certified Cloud Practitioner and start with Solution Architect Associate or Developer Associate certification. In below diagrams, I have denoted Cloud Practitioner as an optional with dotted border.
Below I have list down the Role and corresponding certification path.

Also if you would like to understand eco system of AI services in AWS , AWS AI Practitioner is good way to start your AI journey.

Below I have list down the Role and corresponding certification path.

Solutions Architect

As a solutions architect you need to design, develop, and manage cloud infrastructure and resources. Collaborate with DevOps to migrate applications to the cloud.

Application Architect

As an Application architect, you need to make cloud native application scalable, reliable, and manageable across the entire enterprise.

Cloud Data Engineer

As a Cloud Data Engineer, you need to use Cloud native services to automate the collection and processing of structured and semi-structured data, and monitor the performance of data pipelines.

Cloud-Native-Software Development Engineer

As a Software Development engineer, you develop, build, and maintain software across various platforms and devices. You need understand about serverless development options, what cloud-native option that can be used effectively for software development, release process.

Systems Administrator

As an Systems Admin you install, upgrade, and maintain computer components and software on Cloud VMs, integrate automation processes. Understand what are the various option on cloud to automate the process of patching, making sure systems are compliant.

Cloud DevSecOps Engineer

As a DevOps engineer you need to design, deploy, and operate large-scale global hybrid cloud environments. Advocate for end-to-end automated CI/CD DevOps pipelines.

Cloud engineer

As a Cloud engineer, you need implement and operate an organization’s networked computing infrastructure. Set up security systems to ensure data safety.

Cloud Security Architect

Design and implement enterprise cloud solutions. Apply governance to identify, communicate, and minimize business and technical risks. Similar to engineer, you should be aware security services.

Network Engineer

Design and implement computer and information networks, including local area networks (LAN), wide area networks (WAN), intranets, and extranets.

MLOps Engineer

Build and maintain AI and ML platforms and infrastructure, ensuring scalability, reliability, and efficiency across diverse environments to support data scientists and engineers. Design, implement, and operationally support AI/ML model activity and deployment pipelines, including continuous integration/continuous deployment (CI/CD) workflows, monitoring, and performance optimization to streamline production-grade machine learning solutions.

Machine Learning Engineer

As ML engineer, you design machine learning systems, models, and frameworks. Build artificial intelligence (AI) systems to automate predictive models.

Data scientist

Develop and implement AI/ML models to solve complex business problems. Train and fine-tune models using large datasets to optimize accuracy and efficiency, ensuring alignment with business objectives. Evaluate model performance through rigorous testing and validation, deploying them into production environments

I hope this blogs guides you deciding which certification path to choose to get desired role.

AWS EBS Encryption Simplified : Protecting Your Cloud Data Effectively

vikasbanage — Thu, 12 Dec 2024 20:31:05 +0000

When it comes to data storage on the AWS Cloud, AWS offers a variety of services tailored to meet different needs. Two of the most widely used are Amazon S3 (for object storage) and Amazon Elastic Block Store (EBS) (for block storage). If you need a block device for mounting on instances, with fast data access and long-term persistence, EBS is the go-to choice.

Amazon EBS is tightly integrated with services like EC2 and RDS, making it a reliable and versatile option for many workloads.

But here’s the question we all need to ask ourselves: What Should We Care About When Storing Data on AWS?

The simple answer: Security.

AWS follows a shared responsibility model:

AWS’s Responsibility: "Security of the Cloud"
Customer’s Responsibility: "Security in the Cloud"

This means that any data we store on AWS services, including EBS, is our responsibility to protect. If an attacker gains access to your AWS environment, unencrypted data can be an easy target.

So, how do we secure data stored on EBS?

EBS Encryption.

Encrypting your EBS volumes ensures that your data is protected at rest. Encryption also secures all backups created from the volume and snapshots copied from it.

In this blog, we’ll explore two key methods for encrypting EBS volumes:

Default Encryption: Encrypt new volumes automatically during creation.
Encrypt Existing Non-Encrypted Volumes: Add encryption to volumes that were initially created without it.

Enable Default encryption

By default, when account gets created this setting is disabled. We can enable this easily by going to EC2 Dashboard.

Go to EC2 Dashboard → Under Account Attribute — Data protection and security → Manage

Once you click on Manage, you should able to enable encryption by just selecting checkbox. One important thing to note here, KMS Key you will be using. Here I have selected default key, but I would suggest to create Customer Managed KMS key, make sure KMS key policy that should give permission to role/user which going to be used by EC2/Application.

Once you enabled this setting, whenever you create EC2 instance it’s corresponding EBS volume will get encrypted with above key.

BUT, what about the volume which are created without encryption ?

Encrypting existing Non-Encrypted Volume

It may happen that while creating EC2 instance or EBS we didn’t created volume with encryption. This should not be a problem if we discovered earlier before attacker or auditor discovers ;)

Encrypting a non-encrypted volume is a five step process:

Take snapshot of non-encrypted volume.
Copy & Encrypt Snapshot
Create volume from Encrypted Snapshot
Stop EC2 instance & Detach non-encrypted volume.
Attached encrypted volume & Start EC2 instance.

As a part of this blog, I have done this process in my test environment. I would highly recommend to performance this steps in your test environment first and test your application. If all things work in test , only then proceed on production.

Let’s start encrypting a non-encrypted volume.

As a part of this blog, I have spin-up EC2 instance, installed apache server on it and added simple HTML page. One thing also note, Availability Zone in which you created EC2 instance. EBS and EC2 are zone specific. You cannot attach EBS from AZ-1 to EC2 instance in AZ-2.

So make sure to note-down the Availability Zone of EC2, EBS , it will be easy in performing steps.

As you can see in above screenshot, the volume which is attached to EC2 instance is not encrypted.

1. Take snapshot of non-encrypted volume
Part of this step, go to EBS volume attached to EC2 instance. You can easily do this by selecting EC2 → Go to Storage Tab →Click on corresponding EBS volume which start with vol-

In this step it also good to note down device name for respective volume, in this case it /dev/xvda

You will get navigated to below page where you can select from Action to Create Snapshot.

You can find snapshot section on left pane of AWS console. Go to Snapshot console, click on arrange by creation date (it will be easy for us to check recent snapshot if you have too many snapshots.). You will find recent snapshot creation in process.

After some time depending on size, you should able see the status Completed and progress Available. It means snapshot created successfully.

2. Copy & Encrypt Snapshot

Once snapshot is created successfully from step one, it’s time to copy snapshot and also encrypt. To do that, in snapshot console, select snapshot we created earlier → Action → Copy Snapshot.

During this copy process, you can also encrypt snapshot. For encryption you can provide default KMS key or customer managed KMS key. Make sure KMS key you will be selecting should have KMS key policy which grant access to corresponding EC2 role/ user who will be access application.

Here I have selected default KMS key which have permission to users and roles belongs this account only.

Click on Copy Snapshot, this operation should take some time to complete. Depending on size, time can vary. After 2 minute (in my case), I was able to see encrypted snapshot available.

3. Create volume from Encrypted Snapshot
Now we have encrypted snapshot available, we can create encrypted volume. Select encrypted snapshot → Action → Create Volume from Snapshot.

You will get pop-up after you click Create Volume from Snapshot.

On first option, you can select Volume time GP2, GP3, io1/2 etc. But I would recommend not to change volume type in this case. Keep the volume same as it was before like if current EC2 is on GP2 keep GP2 only, if it is on GP3, keep GP3 only. Our goal here is to encrypt volume only, not to play with performance or other parameters of volume. So if size was 8 GiB, keep 8GiB only. In your case if it is 100GiB, keep it that only.

So jumping on to availability zone, make sure to select correct availability zone. If your current EC2 instance is in us-east-1a, you select us-east-1a. In my case I have EC2 instance in us-east-1f so I selected here us-east-1f.

Click on Create Volume. Volume should be available quickly under Volume console. As you can see, Volume state is Avaialble, it means it is not attached to any instance and the one above which is In-use attached to EC2 instance but it is not encrypted.

So our next step is to attached this encrypted volume to EC2 instance. But before that we need to detach Non-encypted volume from EC2.

4. Stop EC2 instance & Detach non-encrypted volume

As this step leads to application downtime or non-avaialblity. I would suggest to have communication to user who access application. Or best way to have high-available application which is behind Load Balancer and do one server at a time.

In this step, we are going to detach non-ecrypted volume. Before that it is recommended to stop EC2 instance first. Go to corresponding EC2 instance → Instance State →Stop Instance.

Once instance stopped, go to corresponding EC2 volume attached to EC2 instance. It should take you to the volume console, select volume → Actions → Detach Volume. You can confirm when pop-up occured.

Once done, volume state should change to available, may be you also need refresh to reflect state.

5. Attached encrypted volume & Start EC2 instance
In this step we will be attaching encrypted volume to EC2 instance.

Select encrypted volume → Actions → Attach volume.

You will get below pop-up.

In instance drop down, select the instance you stopped in pervious step, you can copy instance-id from EC2 console and just search if there are too many instance in drop-down. If you are not able to see instance, check volume availability zone, it may happen EC2 and EBS volume you created from snapshot are in different AZs.

In device name filed, make sure to enter same name that you have noted down in step-1. In my case it is /dev/xvda

Click on attach volume.

Once volume is attached. Volume state will change In-Use.

Go to EC2 console and start EC2 instance. If the instance is not started, check have you mentioned correct device name as before.

Wait of system checks to finish.

Try accessing your application. In my case I was successfully able to access application 🚀🚀🚀

I hope you found this blog useful. Happy Cloud Computing 🚀

Protect Sensitive Data on AWS: A Beginner’s Guide to Amazon Macie

vikasbanage — Wed, 11 Dec 2024 20:41:49 +0000

The exponential growth of data has enabled businesses to innovate more in their products and services, making them more personalized. As organizations adopt Cloud technology like Amazon Web Services (AWS) to modernize their data capabilities and innovate around it, they face the complexity of managing vast information spread across multiple AWS accounts. Each account corresponds to distinct business units and use cases, adding layers of complexity not only in volume but also in the sensitivity and diversity of the data involved.

This involvement ranges from health information (PHI) and payment card industry (PCI) data to personally identifiable information (PII) and proprietary organizational intellectual property. A data breach involving this information can cause significant financial and reputational losses. Therefore, identifying and safeguarding this sensitive data scattered across accounts becomes one of the critical challenges.

In AWS, Amazon Macie emerges as a critical solution in addressing this challenge. Macie plays a pivotal role in answering below fundamental questions that are essential for robust data security and compliance

- Understanding Data Storage: What data do I have in my S3 buckets, and where is it located?

- Assessing Data Exposure: How is my data being shared and stored, and is it publicly or privately accessible?

- Near Real-Time Data Classification: How can I classify my data in near real-time to ensure constant vigilance?

- Identifying Sensitive Information: What PII or PHI might be exposed publicly, and how can I mitigate this risk?

- Automating Compliance and Security: How do I build and implement workflow remediation to meet my specific security and compliance needs?

What is Amazon Macie?

Amazon Macie is a data security service that uses machine learning and pattern matching to identify sensitive data in Amazon S3, offering insights into security risks and automated protection.

It continuously evaluates S3 buckets using built-in & custom criteria for potential data security or privacy concerns providing findings and detailed statistics for informed decision-making.
Macie integrates with Amazon EventBridge and AWS Security Hub, enhancing its capabilities for monitoring and remediation of data security issues.

Use Cases of Amazon Macie

- Compliance Monitoring and Reporting: Organisations subject to regulations like GDPR, HIPAA, or PCI-DSS can use Macie to automatically discover and classify sensitive data, ensuring compliance by identifying where this data resides and how it’s being used or accessed.

**- Intellectual Property Protection: **Companies can leverage Macie to detect and protect intellectual property stored in S3 buckets, ensuring that proprietary information is not inadvertently exposed or accessed by unauthorised users.

- Mergers and Acquisitions (M&A) Data Security Assessmentss: During M&A activities, Macie can be used to quickly assess the data security posture of acquired or merging entities, identifying sensitive data and ensuring that it complies with corporate policies and regulations.

- Educational Institutions Protecting Student Information: Schools and universities can use Macie to safeguard student records and sensitive information, ensuring compliance with education-related privacy regulations.

- Healthcare Data Management: Healthcare organisations can employ Macie to secure patient data, classifying and protecting health information in accordance with HIPAA and other health data protection standards.

Getting Started with Amazon Macie

Enabling Amazon Macie is a very easy task. Amazon Macie is a regional service, so you need to make sure you select the respective region as per your need from the top left corner in AWS Console.

In order to get started with Amazon Macie, AWS Console Mavie Getting Started → Enable Macie.

In multi-account environments, you can monitor Macie’s usage across your organisation in AWS Organizations through the usage page of the delegated administrator account. For this blog, I’m using a single account set up.

Part of pre-requisite, we need to create S3 bucket to store results. For every object analysed, Macie logs details in ‘sensitive data discovery results,’ including objects it couldn’t analyse due to errors. These results are stored for 90 days, but for longer retention, you can configure Macie to save these results in an S3 bucket.

Just to understand how macie shows findings, you can generate sample findings in Macie. Go to Macie Console → Setting → Generate Sample finding

To view finding, you can go to the Finding page on the same console.

As you can see, in above screenshots Macie generates sample findings related to financial, personal data. You can drill more by selecting specific to finding, it will show bucket, total number of sensitive data, type of information.

Let’s create job now.

You can create a custom job where you can define specific bucket, criteria. Go to Job → Create Job

Select bucket to scan → Refine scope, you can select frequency of job, type of files to scan. For demo purposes, I choose One time job. But depending on your requirement we can schedule a job Daily, weekly and monthly.

In this job, I will be checking the date of birth, I’m choosing a custom managed identifier – “DATE_OF_BIRTH”. So this job should be able to detect these keywords in files : bday,b-day, birth date, birthday, date of birth, dob

I created the sample file with below content and uploaded it to the bucket.

Date of Birth: 1961-04-21
Future Date: 2024-03-04
--------------------
date of birth: 1912-01-20
Future Date: 2024-03-06
--------------------
dob: 1956-05-25
Future Date: 2024-03-16

This One-Time job will get executed once you finish with creation. However, you can also execute this job any time after creation. In few minutes, Macie shows below findings

Above we just see the example of Custom Managed Identifier – date of birth. These identifiers are defined by AWS. You can find a list here.

But let’s say you want to define some customer criteria which is not in the AWS list, you can do it using Custom Data Identifier. With custom data identifiers, you can define detection criteria that reflect your organisation’s particular scenarios, intellectual property, or proprietary data—for example, employee IDs, customer account numbers, or internal data classifications.

Macie Benefits

Compliance Assurance: With its capability to discover and classify sensitive data according to various regulatory standards, Macie assists organisations in meeting compliance requirements for regulations such as GDPR, HIPAA, and PCI-DSS, thereby mitigating the risk of compliance-related penalties.
Automated monitoring and actions: Amazon Macie seamlessly integrates with other AWS services like Amazon EventBridge, AWS Security Hub, and AWS Lambda, facilitating the creation of automated workflows which enables quick response and remediation actions.
Global Data Visibility: It provides organisations with a unified view of their sensitive data across multiple AWS regions and accounts, enhancing the ability to manage data security on a global scale. This visibility is crucial for multinational companies dealing with data residency and sovereignty issues.

Macie pricing

Amazon Macie uses a usage-based pricing model, charging based on the volume of data processed for sensitive data discovery.
First-time users receive a 30-day free trial.
During the free trial, Amazon Macie provides an estimate of monthly costs after the trial ends.
Costs can be controlled by limiting the amount of data scanned, such as:
- Excluding CloudTrail logs from scans.
- Focusing on files with specific extensions.
- Scanning files based on tags.

Conclusion

In a world where data breaches can lead to significant financial and reputational losses, the role of Amazon Macie in safeguarding sensitive data is invaluable. Macie brings benefits that enhance an organisation’s security posture. I hope this blog gives you a good understanding of AWS’s data security service and its importance.