DEV Community: Lucian Patian

AWS SES emails not reaching your business inbox?

Lucian Patian — Tue, 24 Mar 2026 14:19:20 +0000

I invested some time in figuring out why my app running on AWS, using SES for sending emails was not working correctly. The strange behaviour was that for Gmail, it was working fine, for my work domain it was not delivered and I wasn't able to see any logs related to this on my corporate email server.

So I started to check the standard stuff like:

SES in prod mode?
SES config, do I have a from address, am I using the correct region
DKIM enabled?
Custom "mail from" domain configured and valid
No errors on the app side
Checked the SES status for bounces and rejects
Destination email address not being on the suppression list

Everything seemed fine but the test emails were not reaching my business email address. So if Gmail gets the test email, it must be "them", right? The next step was to look into the SPF records, which is a DNS record that tells receiving mail servers "these are the only servers allowed to send email on behalf of our domain." If a server isn't on the list, the email gets flagged or rejected.

The parent domain had a strict policy:

v=spf1 mx ip4:x.x.x.x include:corporate-spf.example.com include:spf.protection.outlook.com -all

That -all at the end means "reject everything not on this list." And SES wasn't on the list, there was no include:amazonses.com anywhere. The sending subdomain didn't have its own SPF record either, so it inherited the parent's strict rules.

I added an SPF record for the subdomain to explicitly allow SES:

v=spf1 include:amazonses.com ~all

But emails still didn't arrive. SPF was part of the problem, but not the root cause.

I needed more visibility to what was happening so I decided to create a SNS topic for SES send & delivery events.

After the next test, the SNS notification came back with this:

{
  "notificationType": "Bounce",
  "bounce": {
    "bounceType": "Transient",
    "bounceSubType": "General",
    "bouncedRecipients": [{
      "emailAddress": "[recipient]",
      "status": "5.3.0",
      "diagnosticCode": "smtp; 553 #5.1.8 Domain of sender address <...@mail.yourdomain.com> does not exist"
    }]
  }
}

It turns out that the corporate mail gateway is rejecting the email because the MAIL FROM domain "does not exist."

So I went back to checking the it's always the DNS:

$ dig MX mail.mydomain.com @8.8.8.8 +short
10 feedback-smtp.eu-central-1.amazonses.com

$ dig TXT mail.mydomain.com @8.8.8.8 +short
"v=spf1 include:amazonses.com ~all"

$ dig A mail.mydomain.com @8.8.8.8 +short
# (empty - no A record)

The issue was that I configured my SES to use MAIL FROM with mail.mydomain.com but that didn't exist, it had no A record. Apparently RFC 5321 says if there's no A record, the domain "doesn't exist" for my corporate mail gateway and the email gets rejected.

The easy fix

I removed the MAIL FROM domain configured in SES so it defaulted to amazonses.com, this way my corporate mail gateway was able to resolve it and emails started to get delivered as expected.

The strange part

The strange part to investigate was the fact that SES was reporting successful deliveries for my test corporate emails.

Here's what actually happens:

SES sends the email to the corporate mail gateway
The gateway accepts the connection (SMTP 250 OK)
Then it runs additional checks: domain existence, SPF, DKIM
It quietly rejects the email with a 553 error

Without delivery notifications, these bounces are invisible. SES counts that initial 250 OK as success. Only when you add SNS notifications does the real story surface.

Lessons learned the hard way

SES stats can be misleading. A gateway can accept an email and then silently discard it. Always enable delivery notifications when debugging. Don't trust the dashboard alone.

RFC 5321 says the MAIL FROM domain must be resolvable. Some strict gateways check for A/AAAA records, not just MX. If you're using a custom MAIL FROM domain with SES, make sure it has an A record. That's the one that I missed...

Import console created resources into CloudFormation

Lucian Patian — Thu, 11 Dec 2025 21:20:28 +0000

This one's for beginners.

Think about each time you created and configured some new AWS services from the console, got really happy about it when you got it right but in the end thought it's going to be a mess to do it all over again.

I'm here to tell you that Infrastructure as a Code will save your ass so don't fight it. It will hurt in the beginning as it's a tough learning curve but it pays off on any given Sunday.

I decided to write this blogpost after migrating 80+ resources across 11 services into CloudFormation because I know it hurts to start something like this and I wanted to give you the push that you needed.

I'm writing IaC on a regular basis in Terraform and CloudFormation, this time I used CloudFormation because I was curious how the import feature works, since I didn't use until now.

Let's talk about the benefits of importing your existing resources into IaC:

version control: track every infrastructure change in git
disaster recovery: recreate your entire infrastructure from code
consistency: deploy identical environments (dev/staging/prod)
change preview: see what will change before applying
team collaboration: code reviews for infrastructure changes
dependency management: CloudFormation prevents breaking changes

There are a few cons on the other hand:

console updates: you can still manually modify resources in the console
drift detection: you need to actively check for drift (manual changes done from the console, not via IaC)

The import process: Overview

create a CloudFormation template describing your existing resources
create an import changeset specifying which resources to import
execute the changeset to bring resources under CloudFormation management

Sounds simple but the devil is in the details.

Lessons learned

1. Never use create-stack for existing resources

aws cloudformation create-stack \
  --stack-name network \
  --template-body file://network.yaml

CloudFormation creates NEW resources instead of importing existing ones. You'll end up with duplicate VPCs, subnets and a mess to clean up.

Always use import changesets:

aws cloudformation create-change-set \
  --stack-name network \
  --change-set-name import-network \
  --change-set-type IMPORT \
  --resources-to-import file://resources.json \
  --template-body file://network.yaml

2. Not all resource types support import

CloudFormation import has limitations. Some resource types simply don't support it.

Resources that DON'T support import:

AWS::Route53::RecordSet - DNS records
route table associations
VPC gateway attachments

Workarounds (any will do):

manage these outside CloudFormation
delete and recreate them (accept brief downtime)
use Terraform instead (supports more imports)

Example: For Route53 records, I deleted existing records and let CloudFormation recreate them. The 5-10 second downtime was acceptable for full IaC coverage.

3. DeletionPolicy is mandatory for imports

Use the DeletionPolicy: Retain to every resource definition or you'll get the Resources must have DeletionPolicy attribute specified in the template error

CloudFormation needs to know what to do if you delete the stack. Retain means "keep the resource even if the stack is deleted."

4. Outputs can't be added during import

Use a two-step process to avoid the You cannot modify or add [Outputs] during import operations error:

Import resources without Outputs section
Update stack to add Outputs after import completes

5. Resource names must match exactly

Your template uses a friendly name but AWS has a different name. Use exact AWS names during import, rename later via stack update to aviod the The Identifier [AlarmName] does not match the identifier value in the template error.

6. Beware of circular dependencies

Security groups often reference each other:

SG-A allows traffic from SG-B
SG-B allows traffic from SG-A

If you use cross-stack references, you create a deadlock:

Stack A exports SG-A, imports SG-B
Stack B exports SG-B, imports SG-A
Neither can be created or updated!

Keep mutually referencing resources in the same stack or as a last resort, use hardcoded IDs for cross references.

7. Start with minimal outputs

Export only what's actually used to aviod dependencies that block stack updates and deletions.

8. Modular stacks are better than monoliths

The monolith approach can break the stack because one change requires updating entire stack, it's difficult to understand and maintain, you have long deployment times:

infrastructure.yaml (5000 lines)
├── VPC
├── Subnets
├── EC2 Instances
├── Security Groups
├── CloudFront
├── Route53
├── S3
└── Everything else

The modular approach:

templates/
├── network.yaml (VPC, subnets)
├── compute.yaml (EC2, security groups)
├── cdn.yaml (CloudFront, WAF)
├── dns.yaml (Route53)
└── storage.yaml (S3)

You update only what changed, have a smaller blast radius, it's easier to understand and you get faster deployments.

Import strategy

Inventory and planning

document existing resources: list everything you need to import
check import support: verify each resource type supports import
identify dependencies: map relationships between resources
plan stack structure: decide on modular vs monolithic approach

Create import templates

start simple: begin with standalone resources (S3, SNS)
add DeletionPolicy: every resource needs DeletionPolicy: Retain
use exact names: match AWS resource names exactly
skip Outputs: don't add Outputs section yet
test in dev: always test import process in non-production first

Execute imports

create import changeset: use --change-set-type IMPORT
review changes: check what will be imported
execute changeset: bring resources under IaC management
verify: confirm resources are managed by CloudFormation

Improve templates

add Outputs: update stacks to add cross-stack exports
replace hardcoded values: Use !ImportValue for dynamic references
document: add comments explaining configurations

Conclusion

always use import changesets, never create-stack
not all resources support import so plan accordingly
DeletionPolicy is mandatory for imports
start modular, multiple focused stacks beat monoliths
minimal exports, avoid dependency hell
test in dev first: always
accept brief downtime for resources that don't support import
document everything, your future self will thank you

Resources

AWS CloudFormation import documentation
Resource types that support import
CloudFormation best practices

CloudFormation change set privilege escalation

Lucian Patian — Fri, 07 Nov 2025 09:36:57 +0000

About a year ago, during a review of our cloud infrastructure, my colleague Martin Birtel from our InfoSec department, discovered that a specific AWS managed policy can give hackers super powers.

The finding was related to the role SecretsManagerReadWrite which was attached to a Lambda responsible for credential management.

This is a great cocktail for an account takeover as you'll see below.

The SecretsManagerReadWrite managed policy includes surprising CloudFormation permissions:

{
  "Version" : "2012-10-17",
  "Statement" : [
    {
      "Sid" : "BasePermissions",
      "Effect" : "Allow",
      "Action" : [
        "secretsmanager:*",
        "cloudformation:CreateChangeSet",
        "cloudformation:DescribeChangeSet",
        "cloudformation:DescribeStackResource",
        "cloudformation:DescribeStacks",
        "cloudformation:ExecuteChangeSet"

      ],
      "Resource" : "*"
    }
  ]
}

It allows the principal (the Lambda execution role in our case) to create and execute CloudFormation change sets. This shouldn't be an issue as long as the Lambda role is not allowed to create all kinds of resources, right?

CloudFormation stacks with attached IAM roles

When you create a CloudFormation stack, you can attach an IAM service role that has permissions to create the defined resources. After stack creation, this service role remains attached and CloudFormation uses it for all future operations, including executing change sets.

If you have permissions to update CloudFormation stacks, you're already in the hacking game. This means that you can use the permissions attached at the stack creation (CloudFormation service role) to make calls to resources in a stack on your behalf.

AWS gives us a heads-up in the official documentation:

When you specify a service role, CloudFormation always uses that role for all operations that are performed on that stack. It is not possible to remove a service role attached to a stack after the stack is created. Other users that have permissions to perform operations on this stack are able to use this role, regardless of whether those users have the iam:PassRole permission or not. If the role includes permissions that the user shouldn't have, you can unintentionally escalate a user's permissions. Ensure that the role grants least privilege.

The escalation path

You have a CloudFormation stack with a high privilege service role attached (let's assume Administrator).
An attacker obtains credentials for a role with SecretsManagerReadWrite policy.
The attacker discovers existing stack names (through enumeration or other means).
The attacker creates a change set targeting the high-privilege stack.
The attacker executes the change set using the stack's service role.

Your CloudFormation stack just created new resources generated by the change set, while your hacker didn't have the permissions to create them. It only had the SecretsManagerReadWrite role which allowed the creation and execution of change sets. Since the CloudFormation service role had Administrator permissions, the change sets could essentially do anything in the account.

Now let's get back to our cheerful Lambda: as it had this SecretsManagerReadWrite attached, it could call the CloudFormation API and create/execute change sets. This means it could inherit the permissions of any CloudFormation stack's service role in the account.

Heads-up: just because a policy is AWS managed, doesn't mean it follows the principle of least privilege for your specific use case.

PS. AWS Support confirmed this is by design.

AWS GameDay: Security

Lucian Patian — Wed, 08 Oct 2025 14:33:02 +0000

Following-up on last year's GameDay which was a real success, we decided to repeat the event, this time with a focus on security.

The event gathered over 100 people in 21 teams from User Groups in Timisoara, Cluj-Napoca, Iasi, Bucharest and Chisinau.

Based on feedback from our meetup participants, we chose security as our focus and we were right, the hype was real.

The entire event was coordinated by our local AWS Solutions Architects, one of them being in each location.

We started with a 10 minute introduction and coordinated all UGs to start exactly at 18:30 so everyone has the same chances.

The teams were of 4-6 people, with quite a lot of students joining, even if they were missing the AWS background. They loved to stay along the experienced participants and to learn by doing.

The event wrapped up at 20:30, with some leading teams completing all tasks within just one hour.

The winning team received VIP tickets for the AWS Community Day CEE happening in Budapest on the 16th of October 2025.

In the networking part, we stayed for pizza and beers while congratulating everyone for participating.

The teams optimized services like IAM Access Analyzer, RDS, Secrets Manager, EventBridge, Config, Lambda, EC2, AMIs. I'm not giving out any other details, as I don't want to spoil the fun.

If you'd like more details about this event, feel free to reach out to me on LinkedIn. If you’re interested in organizing a similar event, make sure to contact your local AWS Solutions Architect or Technical Account Manager.

Copilot vs. Amazon Q: the Claude Sonnet story

Lucian Patian — Thu, 02 Oct 2025 13:32:40 +0000

I've been using Copilot and Amazon Q daily for the past few months, both having the Claude Sonnet 4 model configured.

I know there are many pros and cons against writing quality code with these tools, so I had to convince myself: should I take the blue pill or the red one.

The projects I've been working on are using CloudFormation and Terraform for IaC and the deployment is done using GitLab pipelines.

In Amazon Q, I've also configured a few MCP servers like awslabs.cfn-mcp-server, awslabs.aws-documentation-mcp-server and terraform. In Copilot I wasn't able to configure any, due to company restrictions.

Now the facts:

1. Creating new repositories from scratch

I decided to use CloudFormation to create a new repo from scratch with Amazon Q and the generated templates needed tweaks in about 5-10% of the code. This was impressive.

On the other hand, when it came to deploying the stacks using GitLab, Amazon Q failed to suggest good validation syntax for things like "pipeline inputs". Every time the build failed with issues that weren't easy to fix, I turned to Copilot to get quick and valid fixes, while Amazon Q was a bit behind.

Each time I asked Copilot to scan the repo and provide improvement suggestions, it gave me a reply that I have a solid and mature solution, nothing major can be improved. On the same code, with the same request, Amazon Q delivered a solid number of suggestions which all made sense and in fact I adopted them.

Important: I initially "trusted" Q to use the agent mode and took things too far by committing and pushing new code to the main branch, without asking me first, while working on a feature branch. So that was the moment I decided that I need to manually approve all file changes...

In the end, I was satisfied with finishing the IaC way sooner than starting from scratch in a manual way.

2. Creating new repositories using as inspiration other working repositories

Simply said, this was a bad decision.

This time I used Copilot to start a new repo using Terraform, while instructing it to get inspiration from another Terraform repo which is deployed in production and has 90% the same resources. For the rest of the 10% resources, I instructed Copilot to check the repo I mentioned above, written in CloudFormation and write the equivalent Terraform code.

The result was a total mess-up. The code I instructed to be used as inspiration from the production repo was ignored, Copilot decided to write from scratch new custom Terraform modules, when the production repo was using the official Terraform ones.

I ended up cleaning and rewriting most of the custom modules, turning them into official ones, all with a lot of headaches and frustration.

I tried using Q to fix the repos Copilot created but there was just too much mess and the suggestions were just adding complexity, when I just needed simplicity by using an existing working repo.

Bottom line is that I found more helpful to start from scratch with a new repo, rather than trying to use existing ones and adapting them to my needs. This is exactly the opposite of what I was expecting so I'm curious how the newly Claude Sonnet 4.5 behaves, I'll test them both soon and update this blogpost with my findings.

If you need to choose from the 2, I would go with Q for IaC templates and code reviews but keep Copilot for GitLab pipeline syntax if needed.

Azure AD says expired, AWS OpenSearch says working: the SAML certificate that refused to die

Lucian Patian — Thu, 07 Aug 2025 12:14:21 +0000

It starts innocently enough. You're auditing certificates in Azure AD when you notice the red flag: "EXPIRED" next to your SAML certificate. Has been for months, apparently.

Your first thought: "oh no, authentication must be broken..."

Your second thought: "wait... why is everyone still logging in?"

Welcome to the SAML certificate twilight zone, where expired certificates live forever and security teams question reality.

The "this can't be right" moment

You dig deeper into your monitoring (because you're responsible like that) and confirm the timeline: AWS OpenSearch SAML certificate expired back in August. It's now well into the future, and yet...

Users are happily logging in.
No error messages anywhere.
No angry Teams/Slack messages from the team.
Your monitoring dashboards show everything green.

So naturally, you do what any rational person would do: you panic and assume you're missing something obvious.

The "let me fix this" disaster

Being the helpful engineer you are, you decide to "fix" the problem. You grab the tenant wide metadata (the one with shiny new certificates valid until 2030) and update your AWS OpenSearch.

Result: instant authentication meltdown.

HTTP/2.0 500
Error: Signature validation failed. SAML Response rejected

Users can't log in. Colleagues start to ping you. You quickly revert to the expired certificate and... everything works again.

Plot twist: The "broken" expired certificate was actually the correct one.

This is where things get interesting. You start digging and discover that SAML certificates live in their own special universe where normal rules don't apply.

SSL Certificate expires: browser throws a tantrum, shows scary warnings, users panic.
SAML Certificate expires: crickets chirping, everything continues working like nothing happened.

The technical reality check

Here's what's actually happening (and why it's both clever and concerning):

Azure/Entra AD: "This certificate is expired, but hey, it still signs things mathematically"
AWS OpenSearch: "I can verify this signature cryptographically, expiration date is just a suggestion"
Your security team: "Why is our compliance dashboard lighting up?"

The vendor validation tour

Microsoft Support, with the enthusiasm of someone explaining why water is wet:

"If your application doesn't have any validation for the certificate's expiration and the certificate matches in both Azure AD and your application, your application is still accessible despite having an expired certificate."

Translation: "Working as intended, deal with it."

AWS Support, not to be outdone:

"AWS IAM does not invalidate SAML Responses that are signed using an expired X.509 certificate. The SAML specification itself does not strictly require expiration date validation."

Translation: "This is how SAML works everywhere, not just here."

Then Microsoft drops this gem in their official documentation (appearing not once, but twice because they really want you to get it):

"If your app lacks certificate expiration validation and the certificate matches both Microsoft Entra ID and your app, it remains accessible. This condition is true even if the certificate is expired."

So it turns out this isn't a bug or an oversight. It's a documented behavior that happens to contradict what most people expect.

The "why should I care?" reality check

You're living in a compliance gray area where:

Technical reality: everything works perfectly.
Audit reality: you're using expired certificates.
Security reality: it's not great, but it's not catastrophic either.

The monitoring mirage

Want to monitor your SAML certificate expiration in AWS? Here's what you get:

CloudWatch metrics for SAML certificates: none
OpenSearch configuration for strict validation: none
Built-in AWS monitoring: none

AWS basically says: "Certificate monitoring is Azure's job, we just validate signatures."

Meanwhile, Microsoft offers email notifications at 60, 30 and 7 days before expiration. Which raises the obvious question: why bother setting up alerts for something that keeps working anyway?

The answer: because your compliance auditor doesn't care about SAML protocol nuances.

The infrastructure reality

If you're managing this with CloudFormation, you're in for extra fun since it only supports hardcoded XML metadata content, no URL references like Terraform has for years.

So while your expired certificate keeps working, you still need to manually update CloudFormation templates every time you renew certificates.

I submitted a feature request in September 2023 for URL support. The GitHub roadmap now shows it as "Shipped" but the AWS documentation hasn't been updated to reflect this.

Here's the thing: your expired SAML certificate working is like finding out your car runs fine without oil changes: technically possible, not recommended and definitely not something you want to explain to your manager.

Follow Microsoft's renewal process - it takes a few minutes and prevents uncomfortable audit conversations.

Nov 2026 update:

I've tried to update an existing Cognito CloudFormation stack using an expired certificate XML and it seems that Cognito checks for the certificate validity - it prompted me with this error:

Resource handler returned message: "Signing certificates are expired (Service: CognitoIdentityProvider, Status Code: 400, Request ID: ) (SDK Attempt Count: 1)" (RequestToken: , HandlerErrorCode: InvalidRequest)

Built entirely with Amazon Q CLI.

AWS Client VPN setup was driving me crazy. So I built the easy button

Lucian Patian — Thu, 24 Jul 2025 13:38:30 +0000

You know that feeling when you decide to try something new? Ever thought about setting up an AWS Client VPN?

First, you're optimistic. You search for blog posts, read the AWS documentation and dive in. Then reality hits.

You need to understand certificate generation. Manual steps pile up. You realize this will be a nightmare to reproduce. You try CloudFormation but get cryptic errors like "route already exists" and you have no idea why.

Eventually, you create all the resources following the documentation step by step. But guess what? The connection still doesn't work.

Plot twist: It doesn't have to suck.

Enter: the "just work already" solution

What if VPN setup was more like ordering pizza and less like performing surgery? One command, actual results:

./scripts/deploy-client-vpn.sh \
    --stack-name "please-just-work" \
    --region "us-east-1" \
    --profile "my-sanity" \
    --vpc-id "vpc-12345678" \
    --subnet-id "subnet-87654321" \
    --vpc-cidr "172.31.0.0/16"

Ten minutes later: You have a working VPN. With certificates. And documentation that actually explains why SSH might not work (spoiler: it's the NAT thing and here's how to fix it).

The plot twist nobody tells you

Here's the kicker that breaks everyone: AWS Client VPN uses NAT routing. Your traffic doesn't come from the VPN client IP range - it comes from your subnet range. So when you configure security groups for "VPN clients", you're configuring them wrong.

The real fix: Your private resources need to allow traffic from your private subnet CIDR used in the Client VPN configuration, not from the VPN client CIDR range.

Most tutorials skip this. Most solutions leave you to figure it out. This one puts it right in the troubleshooting guide with the exact commands to fix it. Because 2 AM debugging sessions are nobody's friend.

What you actually get

The reality check: no more certificate generation mysteries or "route already exists" CloudFormation failures.
The time saver: 4 hours of frustration → 10 minutes of deployment.
The bonus: complete cleanup command (because nobody remembers how to tear things down properly).
The relief: works in any AWS account - personal, corporate or that weird sandbox environment.

Perfect for when you need secure access yesterday, not next week after you've become a networking wizard.

The Easy Button

Grab it here: github.com/lucianpatian/aws_clientvpn

Because sometimes you just want the VPN to work so you can get back to the actual work. Revolutionary concept, I know.

Built entirely with Amazon Q CLI.

How to share SSM Parameters across AWS accounts easily

Lucian Patian — Thu, 17 Apr 2025 13:24:11 +0000

Managing configuration settings for multiple AWS accounts might seem overwhelming but it’s actually easy using AWS Systems Manager (SSM) Parameter Store. You can keep all your configuration data in one place and share it safely with other accounts using AWS Resource Access Manager (RAM).

What are Shared Parameters?

Think of shared parameters as a way to avoid the hassle of duplicating configuration data across accounts. Instead of managing the same values separately, you store them centrally in AWS Parameter Store and share them with other accounts that need access. It’s efficient, secure and saves you time.

AWS RAM makes it even easier by letting you:

Pick what to share (like an SSM parameter).
Decide who gets access (specific accounts, groups or entire AWS Organizations).
Set permissions (read-only access).

I began using SSM Parameters to solve a challenge with Route 53 NS entries for DNS delegation between AWS accounts. One account kept recreating its Route 53 zone files via IaC, causing the NS values to change frequently. To automate fetching the updated values and keep DNS delegation intact, I turned to shared parameters in SSM Parameter Store and it was the perfect solution.

Before you start - what you need

Advanced Tier parameters only: You can only share Advanced Tier parameters, not Standard ones.
Encryption for SecureString: If the parameter is encrypted, it must use a customer-managed KMS key. AWS-managed keys won’t work, so make sure to share the KMS key separately.
AWS Organizations Enabled: If you’re sharing within an AWS Organization, ensure sharing is enabled in AWS RAM.

Step-by-step example: sharing and accessing a Parameter

Step 1: Create the Parameter in the Source Account

In the source account, create an Advanced Tier parameter in AWS Systems Manager Parameter Store. You can do this via the console, CLI or SDKs.
Important: Standard parameters cannot be shared between accounts, so make sure the parameter is Advanced Tier

Step 2: Share the Parameter with AWS RAM

Run the following command in the source account to share the parameter with the target account:

aws ram create-resource-share \
    --name "MyParameter" \
    --resource-arns "arn:aws:ssm:REGION:SOURCE_ACCOUNT_ID:parameter/NAME_OF_PARAMETER" \
    --principals "TARGET_ACCOUNT_ID"

Replace the placeholders as follows:

REGION: The AWS region where the parameter exists.
SOURCE_ACCOUNT_ID: The ID of the AWS account where the parameter exists.
NAME_OF_PARAMETER: The name of the parameter you want to share.
TARGET_ACCOUNT_ID: The ID of the AWS account you’re sharing the parameter with.

Step 3: Accept the Share in the Target Account

Now, log in to the target account (the account you shared the parameter with). Here’s what you need to do:

Open the AWS RAM Console.
Go to the Pending Invites section.
Find the invite for the resource share and click Accept.

Note: This step is essential. The target account won’t be able to access the parameter until the invitation is accepted.

Step 4: Find the ARN of the Shared Parameter

In the target account, use the following command to get a list of shared resources:

aws ram list-resources --resource-owner OTHER-ACCOUNTS

This command will return a list of shared resources, including the ARN of the shared parameter. Look for the ARN of the parameter. It will look something like this:

arn:aws:ssm:REGION:SOURCE_ACCOUNT_ID:parameter/NAME_OF_PARAMETER

Important: The aws ssm describe-parameters --shared command will not work for shared parameters, so always use aws ram list-resources.

Step 5: Retrieve the Parameter Value

Now that you have the ARN of the shared parameter, you can use it to retrieve the parameter value. Run this command:

aws ssm get-parameter --name ARN_OF_SSM_PARAMETER

If the parameter is encrypted as a SecureString, you’ll need to include the --with-decryption flag:

aws ssm get-parameter \
    --name ARN_OF_SSM_PARAMETER \
    --with-decryption

Replace ARN_OF_SSM_PARAMETER with the ARN you identified in Step 4.

Points to keep in mind

Sharing SecureString Parameters:

If the parameter is encrypted with a SecureString, make sure the KMS key used to encrypt the parameter is also shared with the target account. You can refer to this guide to learn how to share a KMS key with another account.

Using the right CLI commands:

The aws ram list-resources command is the best way to find the ARN of a shared parameter. Other commands like aws ssm describe-parameters --shared may not work as expected.

Who can use Shared Parameters?

When you share a parameter, the target account gets read-only access. This means they can:

View the current value of the parameter.
Access historical values (if allowed by permissions).

They cannot update, delete or re-share the parameter.

Works great with:

Doesn’t work with:

Run Command in Systems Manager.
CloudFormation dynamic references.
Environment variables in CodeBuild or App Runner.
Secrets in ECS.

How Much Does It Cost?

Source Account (Owner): pays for parameter storage (e.g., $0.05/month per Advanced Tier parameter).
Target Account (Consumer): pays for API calls to access the shared parameter (e.g., $0.05 for 10,000 API requests).

Both accounts have independent usage limits, so one account’s usage won’t impact the other.

Sharing parameters in AWS Parameter Store is an easy way to manage configurations securely and save time. Learn more from the AWS documentation

Centralized Root Access in AWS: A Game-Changer for LandingZone security

Lucian Patian — Fri, 20 Dec 2024 16:20:16 +0000

One of the standout features announced by AWS in 2024 is the ability to control root access keys for all accounts within the Landing Zone using the IAM service. This centralization offers significant benefits for managing security at scale.

Centralized Root Access

By centralizing root access, you can effectively remove and prevent root user credential recovery and access across all accounts enrolled in the Organizations service. This means:

Deleting Root User Credentials: you can remove root user passwords, access keys, signing certificates, and deactivate and delete multi-factor authentication (MFA). New accounts created in Organizations will have no root user credentials by default, enhancing security.
Performing Privileged Tasks: some tasks require root user credentials, but these can now be managed by the management account or a delegated IAM administrator. This allows for secure, controlled access to perform necessary actions without compromising security.

Enabling Account Recovery

If root user credential recovery is needed, the Organizations management account or a delegated IAM administrator can enable the Allow password recovery privileged task. This allows the person with access to the root user email inbox to reset the password and recover credentials. It's recommended to delete root user credentials once the required task is completed to maintain security.

Short-Term Privileged Sessions

AWS now allows for short-term privileged sessions, giving temporary credentials to perform specific root user tasks on member accounts. These sessions enable actions such as:

Deleting misconfigured Amazon S3 bucket policies
Deleting misconfigured Amazon SQS queue policies
Managing root user credentials for member accounts

However, one feature I would love to see is the ability to update unmanageable KMS policies directly. Currently, administrators must log in as the root user, create a support ticket, and have AWS Support grant temporary permissions. For instance, if a key policy grants access to only one user and that user is deleted, the key becomes unmanageable, requiring AWS Support intervention.

Since I was excited to give it a try and remove all the console credentials and the MFA devices registered, I looked into how to create a bash script using AWS CLI to go over all accounts in my organization, check if there's a Console Password set and remove it. The same goes for the MFA Devices for each root user.

The bash script can be found here. It iterates over all accounts in the organization, assumes the IAMDeleteRootUserCredentials
role in each child account, checks if there's any login profile and deletes it for the root user using the delete-login-profile command. Next, it lists the MFA registered devices for the root user using the
list-mfa-devices command and deactivates them using the
deactivate-mfa-device command. Interestingly, you don't need to delete the existing MFA devices; you need to deactivate them. This isn't specified in the official documentation, or I missed it.

The script uses the credentials from the organization account as initial credentials to further assume the IAMDeleteRootUserCredentials role in each child account. If you manage the root credentials for your company, you probably know what I'm talking about.

To make granular changes, I've used the OU ID parameter (PARENT_ID) so you can start with your sandbox and non-prod accounts. This is the only thing you need to change in the script before running it.

Another thing worth mentioning is that the aws sts assume-role command needs the region specified, as STS uses regional endpoints. Also, the --task-policy-arn option has a strange syntax: "arn=arn:aws:iam::aws:policy/root-task/IAMDeleteRootUserCredentials". Notice it starts with arn=arn, which is a terrible way of doing it. I honestly have no idea why the service team implemented it this way.

If your teams are using GuardDuty for monitoring suspicious activity in their accounts, you might want to give them a heads-up as the deletion of the root credentials will trigger an alert of Policy:IAMUser/RootCredentialUsage type which looks like this:

The API GetAccountSummary was invoked using root credentials from IP address

This feature greatly enhances the ability to manage and secure AWS environments effectively, ensuring that root access is controlled and monitored at all times.

Have you tried this new feature? What do you think about it?

How to Create a New Entra ID Enterprise Application and Configure Custom Attributes for SAML Login for AWS Cognito

Lucian Patian — Tue, 15 Oct 2024 09:51:23 +0000

When integrating Entra ID (formerly Azure AD) with AWS Cognito for SAML login, it's important to use a unique attribute to identify users. In this guide, we'll walk you through the steps to create a new Enterprise Application in Entra ID and configure a custom attribute named user.objectid. This attribute ensures that user identities remain consistent even if other attributes, such as last names, change.

Why use `user.objectid`?

The user.objectid attribute is unique to each user in Entra ID and does not change, even if other user attributes are updated. This is particularly important for scenarios where an employee changes their last name, as other attributes will be updated with the new value. Using user.objectid prevents the creation of a new local user in your application and ensures that existing user data is preserved.

Steps to Create a New Enterprise Application in Entra ID

Create a New Application

Navigate to the Azure Portal
- Open the Azure Portal and go to the Entra ID service.
Create a New Application
- Go to Enterprise applications and select New application.
- Choose Create your own application.
- Select the option to Integrate any other application you don’t find in the gallery (Non-gallery).
- Type the name of your new application and create it.

Configure Single Sign-On (SSO) login

Create the connection between Entra ID and your application by setting the login URL and the identity of your application.

Select Single Sign-On Method
- In your Enterprise application, select the Single Sign-On option from the left menu.
- Click on SAML as the single sign-on method.
Edit Basic SAML Configuration
- Edit the Basic SAML Configuration.
- Add the Identifier (Entity ID) and Reply URL (Assertion Consumer Service URL).
  - The Identifier (Entity ID) should follow the format: urn:amazon:cognito:sp:cognito_userpool_id.
  - The Reply URL (Assertion Consumer Service URL) should follow the format: https://cognito_domain_url/saml2/idpresponse.
  - Save the changes to the Basic SAML Configuration.

Configure the User Access for SSO login

Assign the users and groups that should have permissions to log in to your application.

Assign Users and Groups
- Select the Users and groups option from the Enterprise application options.
- Click on Add user/group.
- Select the users and/or groups who should have access to your application.
- Confirm your selections and save.

Configure User Attributes & Claims for SSO Login

Configure which Entra ID attributes should be used to log in to your application.

Edit User Attributes & Claims
- From the Single Sign-On option for your Enterprise application, edit the User Attributes & Claims.
Set Unique User Identifier
- Select the Unique User Identifier (Name ID) claim to edit it.
- In the Source attribute, set the value to user.objectid.
- Save the changes.

Update the AWS Cognito userpool

Once you have defined all the claim mappings on the Entra ID side, it is time to connect the dots on AWS's side.

Retrieve SAML Federation Metadata

This is the intermediate step between configuring Entra ID and Cognito.

Get the Federation Metadata URL
- In your Entra ID Enterprise application, navigate to the Single Sign-On section.
- Locate the App Federation Metadata Url.
- Copy this URL, as it will be needed in AWS Cognito.

Enable the IdP

After all configurations are done on Entra ID side, you need to update the configuration in Cognito.

Enable the IdP
- In AWS Cognito, select the User Pool and go to the Sign-in experience tab.
- Under the Federated identity provider sign-in section, click on Add identity provider.
- Select the SAML type.
- Enter a name under Provider name.
- Under Identifiers, add the IdpIdentifier value.
- Use the recommended setting Require SP-initiated SAML assertions for the IdP-initiated SAML sign-in setting.
- Enter the metadata document endpoint URL saved previously.

This is a better solution than uploading the XML file because Cognito refreshes the metadata every 6 hours or before the metadata expires. This way, you don’t have to manually refresh the metadata XML every time the Entra ID SSL certificates expire or any other change occurs on the Entra ID side that would impact the federation authentication.

Configure Attribute Mapping

Configure the attributes that are stored in Entra ID and are mapped via the SAML schema in AWS Cognito. Here is a copy-and-paste friendly table for easier usage:

SAML Attribute	User Pool Attribute
http://schemas.xmlsoap.org/ws/2005/05/identity/claims/upn	Profile
http://schemas.xmlsoap.org/claims/CommonName	Preferred User Name
http://schemas.xmlsoap.org/ws/2005/05/identity/claims/givenname	Given Name
http://schemas.xmlsoap.org/ws/2005/05/identity/claims/surname	Family Name
http://schemas.xmlsoap.org/ws/2005/05/identity/claims/emailaddress	Email

Enable the External IdP for App Clients

Now that you have an IdP using the Entra ID configuration, you need to assign it to your application created in the Cognito userpool.

Enable the IdP for App Clients
- In AWS Cognito, navigate to the App integration tab, App client list section.
- Select the App client you want to configure and edit the Hosted UI section.
- From the Identity providers dropdown, select your newly created IdP (e.g., EntraID) and save the changes.

Test the Configuration

Click on the View Hosted UI button to quickly test your changes

Final Steps

After a user has successfully authenticated via the external IdP, it will automatically be created in your Cognito userpool with the "Enabled" status. The confirmation status will be set to EXTERNAL_PROVIDER.

The user attribute identities will store the metadata relating to the external IdP that “owns” this identity. This includes the user’s ID in the external IdP’s attribute, in our case, the "Identifier (Entity ID)".

These fields will be updated on each successful authentication, so you can rely on the fact that the fields you receive via JWT attributes will be up-to-date.

Conclusion

By following these steps, you will have a fully functioning solution offering federated authentication with an external Entra ID. This setup ensures that user identities remain consistent and up-to-date, even when user attributes change.

AWS GameDay: Frugality Fest

Lucian Patian — Fri, 14 Jun 2024 14:32:42 +0000

AWS GameDay is an engaging, hands-on learning event where participants tackle real-world technical challenges using AWS solutions in a team-based environment. Unlike traditional workshops, GameDays are open-ended and encourage creative problem-solving.

To foster collaboration among AWS User Groups in Romania, we organized a GameDay event focused on building cost-effective applications and mastering cost-efficiency strategies. We chose the "Frugality Fest" theme, a trending topic for 2024, because it aligns well with our evening meetup schedule, typically starting at 18:30. This ensured the event wouldn't stretch late into the night, making it convenient for participants who join after their workday.

Each AWS Usergroup from Romania (Timisoara, Cluj-Napoca, Bucuresti, Iasi) and Moldova hosted on-site events coordinated remotely by AWS Romania. Each location had a dedicated AWS Romania representative to help with organization and coordination.

Teams of four participated, with some pre-formed and others assembled ad-hoc. The event kicked off with a 30 minute introduction covering the event's objectives, the scenario and the rules.

At 18:30, the game began and all 17 teams accessed their resources. Each venue featured a live scoreboard displayed prominently, updating in real-time.

The game ran for 2.5 hours, filled with intense communication, laughter, frustration and calls for help. After the clock stopped, we captured a screenshot of the final scoreboard. The top three teams were the most experienced but everyone enjoyed the event and eagerly asked about the next GameDay.

While I won't spoil the specific tasks, they involved AWS services like EC2, RDS, S3, VPC and CloudWatch. A tip from the winners: prioritize database optimization!

Event photos:

Use AWS StepFunctions for SSM Patching Alerts

Lucian Patian — Thu, 11 Apr 2024 07:42:16 +0000

In this blog post we'll explore how to use AWS Step Functions and SSM Patch Manager to monitor the patch compliance status of EC2 instances and send alerts, reducing manual tracking and enhancing the security of our cloud environment.

AWS Step Functions is a service that doesn't require a server to run. It allows us to connect with Lambda functions and other services to construct important business applications.

The service is built around the principle of linked tasks put together in a workflow called "state machine". A task is able to invoke other AWS services or more recently third-party APIs.

AWS Systems Manager (SSM), which includes the Patch Manager feature, provides a unified interface for managing your AWS resources, including the ability to automate patching for EC2 instances which can make things easier for us.

However, there are instances that need a reboot so the EC2 patch is completely done. We don't want to restart them automatically, especially when they're running critical services like databases. In this case, we prefer to choose when to restart.

To keep track of the EC2 instances that aren't fully compliant in the SSM patch report, we need alerts.

My goal was to send alerts to a Microsoft Teams channel, listing the EC2s that aren't compliant and need additional actions like rebooting. Initially, I used a Lambda function to do this but I didn't want to manage its dependencies over time so I switched to using Step Functions, taking advantage of its new feature that supports HTTPS endpoints.

Overview of the State Machine

The entire process is initiated by an EventBridge rule, which acts as a trigger for the Step Function state machine.

The state machine begins by identifying all currently active EC2 instances in the AWS account. It then retrieves all the instance IDs and filters them based on parameters such as ComplianceType=Patch and Status=NON_COMPLIANT.

Next it determines if there are any instances that need review. If not, the state machine will skip to the end and stop. To do this, we use a task that counts the number of instances in the list from the previous step. If the count is more than zero, indicating that there are instances requiring attention, the state machine continues to filter these instances by their tags. This information is then used to format a message sent to a Microsoft Teams channel, which includes the names and IDs of the EC2 instances that need our attention.

In the end we call the 3rd party API to send the formatted message to the Microsoft Teams channel.

Full steps description

DescribeInstances: starts the process by identifying all currently active EC2 instances within the AWS account.
ExtractInstanceIDs: retrieves all the instance IDs from the previously fetched list of EC2 instances.
FetchInstanceComplianceData: filters the instances based on the ComplianceType=Patch and Status=NON_COMPLIANT parameters.
CalculateArrayLength: calculates the size of the list of non-compliant instances.
CheckIfInstancesFound: checks if the size of the list is greater than zero (indicating that there are non-compliant instances) or not. If no non-compliant instances are found during this step, the state machine skips to the end state and stops.
DescribeTagsForFilteredInstances: if there are non-compliant instances, this step fetches the tags for these instances.
PrepareNonCompliantInstanceList: prepares a list of non-compliant instances along with their names and IDs.
CallThirdPartyAPI: formats the message with the non-compliant instances' information and sends it to a Microsoft Teams channel.

Additional Configuration Details

This state machine can be used to send the message to any communications tool like Slack, MS Teams or to a ticketing system.

For MS Teams, the endpoint URL needs to be encoded so the "@" needs to be replaced with "%40" or you can use a URL shortener service.

An HTTP Task requires an EventBridge connection, which securely manages the authentication credentials of an API provider. A connection specifies the authorization type and credentials to use for authorizing a third-party API.

In our case we are just sending a message/payload to an external URL without the need of authentication but in order to use the StepFunction HTTP Task, we need to create this connection. When creating the connection, as a requirement, you also create an AWS Secret used for authentication. Again, since there's no need to authenticate to the MS Teams channel, the Secret values contain the keyname of the API and the secret is the ARN of the EventBrdige API connection:

Infrastructure as Code

The entire PoC was done using the AWS Console but since we are living in the age of automation, I wanted to have an easy and repeatable way of deploying the solution.

In the past weeks, the Cloudformation service team announced the new IaC generator (infrastructure as code generator) which must be one of the most desired features for years now so I definitely wanted to give it a try.

It turns out that I was able to get the Cloudformation template for all the needed resources pretty easy. The hardest thing was to select from a huge dropdown list all the resources involved in my scenario and to make sure I don't leave out any. After the template was generated, inside the StepFunction JSON definition, it was a bit difficult to replace the hardcoded values with parameters. Now it seems like a piece of cake.

If you want to use this solution, checkout the Github repo which includes the entire Cloudformation stack needed for deployment.

During the initial stages of this PoC, I encountered difficulties with the CallThirdPartyAPI task so I asked around for guidance other AWS CommunityBuilders in the dedicated Slack space and got almost instant help from Benoît Bouré, Jimmy Dahlqvist and Andres Moreno. Chapeau bas!

DEV Community: Lucian Patian

AWS SES emails not reaching your business inbox?

The easy fix

The strange part

Lessons learned the hard way

Import console created resources into CloudFormation

The import process: Overview

Lessons learned

1. Never use create-stack for existing resources

2. Not all resource types support import

3. DeletionPolicy is mandatory for imports

4. Outputs can't be added during import

5. Resource names must match exactly

6. Beware of circular dependencies

7. Start with minimal outputs

8. Modular stacks are better than monoliths

Import strategy

Conclusion

Resources

CloudFormation change set privilege escalation

This is a great cocktail for an account takeover as you'll see below.

CloudFormation stacks with attached IAM roles

The escalation path

Heads-up: just because a policy is AWS managed, doesn't mean it follows the principle of least privilege for your specific use case.

AWS GameDay: Security

Copilot vs. Amazon Q: the Claude Sonnet story

Now the facts:

1. Creating new repositories from scratch

2. Creating new repositories using as inspiration other working repositories

Azure AD says expired, AWS OpenSearch says working: the SAML certificate that refused to die

The "this can't be right" moment

The "let me fix this" disaster

The technical reality check

The vendor validation tour

The "why should I care?" reality check

The monitoring mirage

The infrastructure reality

Nov 2026 update:

AWS Client VPN setup was driving me crazy. So I built the easy button

Enter: the "just work already" solution

The plot twist nobody tells you

What you actually get

The Easy Button

How to share SSM Parameters across AWS accounts easily

What are Shared Parameters?

Before you start - what you need

Step-by-step example: sharing and accessing a Parameter

Step 1: Create the Parameter in the Source Account

Step 2: Share the Parameter with AWS RAM

Step 3: Accept the Share in the Target Account

Step 4: Find the ARN of the Shared Parameter

Step 5: Retrieve the Parameter Value

Points to keep in mind

Sharing SecureString Parameters:

Using the right CLI commands:

Who can use Shared Parameters?

Works great with:

Doesn’t work with:

How Much Does It Cost?

Centralized Root Access in AWS: A Game-Changer for LandingZone security

Centralized Root Access

Enabling Account Recovery

Short-Term Privileged Sessions

How to Create a New Entra ID Enterprise Application and Configure Custom Attributes for SAML Login for AWS Cognito

Why use user.objectid?

Steps to Create a New Enterprise Application in Entra ID

Create a New Application

Configure Single Sign-On (SSO) login

Configure the User Access for SSO login

Configure User Attributes & Claims for SSO Login

Update the AWS Cognito userpool

Retrieve SAML Federation Metadata

Enable the IdP

Configure Attribute Mapping

Enable the External IdP for App Clients

Test the Configuration

Final Steps

Conclusion

AWS GameDay: Frugality Fest

Use AWS StepFunctions for SSM Patching Alerts

Why use `user.objectid`?