DEV Community: Alejandro Velez

Modernizing Cross-Account Networking: From VPC Peering Complexity to 5 VPC Lattice Resources - NetDevOps with AWS

Alejandro Velez — Sat, 13 Jun 2026 21:11:11 +0000

AWS Level: 400 — Advanced | Multi-account networking, VPC Lattice Resource Configurations, IaC orchestration with Terragrunt

🎯 The Challenge

Scenario

Suppose you are a Cloud Architect or DevOps Engineer at an enterprise organization, and you receive this mission: expose an Aurora PostgreSQL database to an external AWS organization — without VPC Peering, without Transit Gateway, without public endpoints, and without deploying a single NLB.

The requirements are clear:

✅ Zero internet exposure — the database must remain in private subnets
✅ Cross-account, cross-organization — the consumer lives in a completely separate AWS org
✅ Infrastructure as Code — everything must be declared, versioned, and reproducible
✅ Network validation — you must prove connectivity works before handing it off
✅ Least privilege — security groups scoped to exact ports and sources
✅ Multi-environment — same code for dev, qa, and prod with different configurations

Welcome to NetDevOps.

Cloud Networking at Scale is Broken (And We Keep Pretending It's Not)

I've been working with multi-account AWS architectures for years, and there's one conversation that keeps coming back like a boomerang: "How do we give that other team access to our database without blowing up our network?"

Every. Single. Time.

And the answers are always the same tired playbook: VPC Peering (good luck with overlapping CIDRs), Transit Gateway (hope you enjoy $0.05/GB bills), PrivateLink (another NLB to maintain). They all work, technically. But they all introduce complexity that compounds over time.

Here's the thing — cloud networking at scale isn't hard because any single concept is difficult. It's hard because everything multiplies. One VPC? Easy. Ten with peering? Fine. Fifty accounts across business units with different compliance requirements, connectivity patterns, and team boundaries? That's where traditional approaches collapse.

I'm talking about:

Security group sprawl where nobody can tell which rules are actually needed
Route table entropy that breaks production with one wrong entry
Cross-account connectivity patterns that multiply with every new team
Connectivity validation that only happens after the incident

This is exactly why I've been investing time in NetDevOps — and today I want to show you how I solved the challenge above.

What is NetDevOps?

Before jumping into the implementation, let me share what NetDevOps means in practice (and why I think every cloud engineer should care).

NetDevOps applies DevOps principles to network infrastructure. That's it. But the implications are massive:

Networks declared as code — not clicked together in a console
Connectivity tested before deployment — not validated after outages
Changes reviewed as pull requests — not approved in 3-hour CAB meetings
Configurations reproducible across environments — not buried in wiki pages that nobody updates

In AWS terms? It means treating VPCs, security groups, VPC Lattice configs, and connectivity paths as first-class IaC artifacts. With real dependency management. With real tests. With real CI/CD.

And the tool I keep reaching for to orchestrate all of this? Terragrunt + OpenTofu. The combination gives me layered dependencies, environment promotion, and DRY configurations without losing visibility into what's actually deployed.

My Approach to the Challenge

So how do we solve this mission? I evaluated the traditional options and rejected all of them:

Approach	Why I Said No
VPC Peering	Non-transitive, CIDR overlap risk, routing table management nightmare
Transit Gateway	Expensive at scale, shared blast radius, route table limits
PrivateLink + NLB	An NLB for every resource? Port management? No thanks.
Database replication	Double cost, consistency challenges, operational overhead
Public endpoint	🙈 Let's not even go there

Then I remembered that AWS had released VPC Lattice Resource Configurations with ARN-based targeting. I'd been keeping an eye on it since re:Invent, and this was the perfect use case to test it.

Spoiler: it worked beautifully. And when I combined it with VPC Reachability Analyzer for automated validation, I had a full NetDevOps pipeline for cross-account connectivity.

The Architecture

This is what we're building. Three accounts, one region, zero internet exposure:

Account	Role	What Lives There
External Account (ours)	Hosts the database + VPC Lattice	VPC, Aurora PostgreSQL, Resource Gateway, Service Network, RAM Share
Network Account	Owns the Lattice mesh	Service Network, Resource Configuration
Workload Account (consumer)	External org that needs DB access	Applications connecting via VPC Lattice endpoint

The key components:

VPC Lattice Service Network — the connectivity mesh
Resource Gateway — network path into our VPC (deployed in DB subnets)
Resource Configuration (ARN-based) — points to the Aurora cluster by ARN
RAM Resource Share — shares the Service Network cross-account
VPC Reachability Analyzer — proves the path works before anyone complains

How I Organized the Code

I'm a firm believer in layered architecture for IaC. If you've read my previous posts on Terragrunt, you know I follow a DDD-inspired approach where each layer has clear boundaries and dependencies flow in one direction:

stacks/
├── foundation/
│   └── network/
│       ├── vpc/                    # VPC + database subnets only
│       └── security-groups/        # SGs for RDS, Lattice RGW, VPC association
├── platform/
│   ├── data/
│   │   └── rds/                    # Aurora PostgreSQL cluster
│   └── network/
│       └── vpc-lattice/            # Service Network + Resource Gateway + RAM
└── observability/
    └── network/
        └── reachability-analyzer/  # Network path validation (the NetDevOps piece!)

Foundation → Platform → Observability. Never backwards. Dependencies are explicit in every terragrunt.hcl.

Why does this matter? Because when someone changes a security group rule, I can immediately see which platform and observability stacks will be affected. No surprises.

Hands-On: Let's Build It

Alright, enough theory. Let's get our hands dirty. I'll walk you through each layer, explain my decisions, and show you the actual code running in this project.

Prerequisites:

OpenTofu or Terraform >= 1.5
Terragrunt >= 0.98
AWS CLI v2 configured with appropriate profiles
An S3 bucket + DynamoDB table for remote state

Step 1: The VPC (Minimal Attack Surface)

Here's something I feel strongly about: if a VPC only hosts a database, it should only have database subnets. No public subnets. No internet gateway. No NAT. Nothing that doesn't need to be there.

# stacks/foundation/network/vpc/main.tf
module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "6.6.0"

  name = "${var.project}-${var.environment}-vpc"
  cidr = var.vpc_cidr

  azs              = var.availability_zones
  database_subnets = var.database_subnets

  create_database_subnet_group       = true
  create_database_subnet_route_table = true
  create_igw                         = false
  enable_nat_gateway                 = false

  manage_default_security_group  = true
  default_security_group_ingress = []
  default_security_group_egress  = []

  enable_dns_hostnames = true
  enable_dns_support   = true

  tags = var.tags
}

I love this. Zero internet exposure by design. The default security group is locked down with empty rules. DNS is enabled because VPC Lattice needs it for resolution. That's it — nothing more, nothing less.

Step 2: Security Groups (This Is Where It Gets Interesting)

Most teams I work with define security groups with CIDR blocks. That works until subnets change, IP ranges shift, or someone adds a new AZ. I prefer security group references — they create a logical coupling that survives all those changes:

# stacks/foundation/network/security-groups/main.tf

# RDS only accepts traffic from the Lattice Resource Gateway
module "rds_sg" {
  source  = "terraform-aws-modules/security-group/aws"
  version = "6.0.0"

  name        = "${var.project}-${var.environment}-rds"
  description = "Security group for RDS Aurora cluster"
  vpc_id      = var.vpc_id

  ingress_rules = {
    postgresql_from_lattice = {
      description                  = "Allow PostgreSQL from VPC Lattice resource gateway"
      from_port                    = 5432
      to_port                      = 5432
      ip_protocol                  = "tcp"
      referenced_security_group_id = module.lattice_resource_gateway_sg.id
    }
  }

  tags = merge(var.tags, { Name = "${var.project}-${var.environment}-rds-sg" })
}

# Resource Gateway can only talk to RDS
module "lattice_resource_gateway_sg" {
  source  = "terraform-aws-modules/security-group/aws"
  version = "6.0.0"

  name        = "${var.project}-${var.environment}-lattice-rgw"
  description = "Security group for VPC Lattice Resource Gateway"
  vpc_id      = var.vpc_id

  egress_rules = {
    to_rds = {
      description                  = "Allow PostgreSQL to RDS"
      from_port                    = 5432
      to_port                      = 5432
      ip_protocol                  = "tcp"
      referenced_security_group_id = module.rds_sg.id
    }
  }

  tags = merge(var.tags, { Name = "${var.project}-${var.environment}-lattice-rgw-sg" })
}

See what's happening here? The Resource Gateway can only reach RDS on port 5432. And RDS only accepts connections from the Resource Gateway. This is defense in depth — even if someone compromises the VPC, lateral movement is blocked by design.

Step 3: Aurora PostgreSQL

Nothing revolutionary here — I'm using the official terraform-aws-modules/rds-aurora module because it handles all the complexity of parameter groups, subnet groups, and monitoring configuration:

# stacks/platform/data/rds/main.tf
module "aurora" {
  source  = "terraform-aws-modules/rds-aurora/aws"
  version = "10.2.0"

  name            = "${var.project}-${var.environment}-aurora"
  engine          = "aurora-postgresql"
  engine_version  = var.engine_version
  master_username = var.master_username

  vpc_id                 = var.vpc_id
  db_subnet_group_name   = var.db_subnet_group_name
  create_security_group  = false
  vpc_security_group_ids = [var.rds_security_group_id]

  storage_encrypted           = true
  apply_immediately           = true
  cluster_monitoring_interval = 60

  instances = var.instances

  backup_retention_period = 7
  deletion_protection     = var.deletion_protection
  skip_final_snapshot     = var.environment != "prod"

  tags = var.tags
}


# -----------------------------------------------------------------------------
# Discover the RDS writer ENI for downstream reachability analysis
# This runs in the same stack where the cluster is created, so ENI always exists
# -----------------------------------------------------------------------------
data "aws_network_interfaces" "rds_writer" {
  filter {
    name   = "group-id"
    values = [var.rds_security_group_id]
  }

  filter {
    name   = "status"
    values = ["in-use"]
  }

  depends_on = [module.aurora]
}

The important detail: create_security_group = false. I pass in the security group from the foundation layer. This keeps ownership clear — the network team controls access rules, the data team controls database configuration.

And here's how Terragrunt wires the dependencies:

# stacks/platform/data/rds/terragrunt.hcl
dependency "vpc" {
  config_path = "../../../foundation/network/vpc"
  mock_outputs = {
    vpc_id                     = "vpc-mock-12345"
    database_subnet_group_name = "mock-db-subnet-group"
  }
  mock_outputs_merge_strategy_with_state = "shallow"
}

dependency "security_groups" {
  config_path = "../../../foundation/network/security-groups"
  mock_outputs = {
    rds_security_group_id = "sg-mock-rds"
  }
  mock_outputs_merge_strategy_with_state = "shallow"
}

Those mock_outputs are critical — they let you run terragrunt plan on individual stacks without deploying everything first. Essential for fast feedback loops.

Step 4: VPC Lattice (Here's Where the Magic Happens)

OK, this is the part I'm most excited about. VPC Lattice with DNS-based Resource Configurations is genuinely elegant. Let me show you:

# modules/vpc-lattice/main.tf

# 1. Create the Service Network (the mesh)
resource "aws_vpclattice_service_network" "this" {
  name      = var.service_network_name
  auth_type = var.service_network_auth_type
  tags      = merge(var.tags, { Name = var.service_network_name })
}

# 2. Associate our VPC to the mesh
resource "aws_vpclattice_service_network_vpc_association" "this" {
  vpc_identifier             = var.vpc_id
  service_network_identifier = aws_vpclattice_service_network.this.id
  security_group_ids         = var.vpc_association_security_group_ids
}

# 3. Deploy the Resource Gateway (network path into our VPC)
resource "aws_vpclattice_resource_gateway" "this" {
  name               = var.resource_gateway_name
  vpc_id             = var.vpc_id
  subnet_ids         = var.resource_gateway_subnet_ids
  security_group_ids = var.resource_gateway_security_group_ids
  ip_address_type    = "IPV4"
}

# 4. Create the Resource Configuration — DNS + custom domain!
resource "aws_vpclattice_resource_configuration" "this" {
  name = var.resource_config_name

  resource_gateway_identifier = aws_vpclattice_resource_gateway.this.id
  custom_domain_name          = var.custom_domain_name

  port_ranges = ["5432"]
  protocol    = "TCP"

  resource_configuration_definition {
    dns_resource {
      domain_name     = var.resource_dns_name  # Aurora cluster endpoint
      ip_address_type = "IPV4"
    }
  }
}

# 5. Associate the resource to the service network
resource "aws_vpclattice_service_network_resource_association" "this" {
  resource_configuration_identifier = aws_vpclattice_resource_configuration.this.id
  service_network_identifier        = aws_vpclattice_service_network.this.id
}

That's it. Five resources and you've exposed an RDS cluster to a service mesh with a custom private DNS name that consumers can use. No NLBs. No PrivateLink endpoints. No route tables.

⚠️ Important lesson learned: VPC Lattice custom_domain_name is only supported on DNS-type and IP-type resource configurations — NOT on ARN-type. If you try type = "ARN" with custom_domain_name, the API returns ValidationException: Unexpected attribute customDomainName for type ARN. That's why we use dns_resource pointing to the Aurora cluster endpoint, which gives us the custom domain capability while still routing to the database correctly.

The custom_domain_name is key here — instead of consumers connecting to an auto-generated Lattice DNS endpoint, they connect using something like db.dev.internal.netdevops.com. This makes migration transparent and application configs clean.

Cross-Account Sharing with RAM (and Resharing!)

In our architecture, we don't share directly with every consumer. Instead, we share the Service Network with a Network Account that acts as a hub — and that account reshares with downstream workload accounts:

External Account → Network Account (reshare enabled) → Workload Accounts

The magic is in allow_external_principals. By enabling it, the Network Account can accept the share and propagate access to downstream consumers. AWS automatically assigns the appropriate VPC Lattice permission to the share:

resource "aws_ram_resource_share" "this" {
  count = var.ram_share_enabled ? 1 : 0

  name                      = var.ram_share_name
  allow_external_principals = true
}

resource "aws_ram_resource_association" "service_network" {
  count = var.ram_share_enabled ? 1 : 0

  resource_arn       = aws_vpclattice_service_network.this.arn
  resource_share_arn = aws_ram_resource_share.this[0].arn
}

resource "aws_ram_principal_association" "this" {
  count = var.ram_share_enabled ? length(var.ram_external_principals) : 0

  principal          = var.ram_external_principals[count.index]
  resource_share_arn = aws_ram_resource_share.this[0].arn
}

⚠️ Gotcha: You might find blog posts suggesting you pass permission_arns with an AllowReshare ARN to enable resharing. In practice, this can fail with InvalidParameterException depending on your region/account. Let AWS assign the default permission automatically — it handles the reshare capability through the RAM console or API when the Network Account accepts and reshares.

Why this pattern? Because in enterprise organizations, the team that owns the database shouldn't need to know who is consuming it. That's the Network Account's job — they control the mesh, they approve consumers, they manage the blast radius. Separation of concerns applied to network sharing.

Step 5: Network Testing as Code (My Favorite Part)

Here's where NetDevOps really comes together. Most teams deploy network infrastructure and then... hope it works. Maybe someone runs a telnet from an EC2 instance to check connectivity. Maybe they wait for an application error.

I wanted something better. I wanted a test that runs as part of the infrastructure deployment and tells me: "yes, traffic can flow from the Resource Gateway to RDS on port 5432."

Enter VPC Reachability Analyzer:

# modules/reachability-analyzer/main.tf

# Define the path we want to test
resource "aws_ec2_network_insights_path" "lattice_to_rds" {
  count = var.enabled ? 1 : 0

  source           = var.resource_gateway_eni_id
  destination      = var.rds_endpoint_network_interface_id
  destination_port = var.destination_port
  protocol         = "tcp"

  tags = merge(var.tags, { Name = "${var.name_prefix}-lattice-to-rds" })
}

# Run the analysis
resource "aws_ec2_network_insights_analysis" "lattice_to_rds" {
  count = var.enabled ? 1 : 0

  network_insights_path_id = aws_ec2_network_insights_path.lattice_to_rds[0].id
  wait_for_completion      = true

  tags = merge(var.tags, { Name = "${var.name_prefix}-lattice-to-rds-analysis" })
}

Notice the var.enabled guard. This is critical — the Reachability Analyzer requires real ENI IDs (not mocks). On first deployment, the upstream stacks (RDS, VPC Lattice) may not have provisioned their ENIs yet. The enabled flag skips creation when ENIs are null, preventing the InvalidParameterValue: eni-mock-xxx is not a supported Source type error you'd otherwise get. On the second apply, real ENI IDs flow through and the analysis runs.

The RDS stack discovers its own writer ENI and outputs rds_writer_eni_id
The VPC Lattice stack discovers its Resource Gateway ENI and outputs resource_gateway_eni_id
The Reachability Analyzer receives both as inputs

This enables one-shot layer-by-layer deployment. During terragrunt plan, Terragrunt provides mock ENI values so the plan succeeds without deployed resources. During apply, real ENI IDs flow through the dependency chain:

# stacks/observability/network/reachability-analyzer/terragrunt.hcl
dependency "rds" {
  config_path = "../../../platform/data/rds"
  mock_outputs = {
    rds_writer_eni_id = "eni-mock-rds-writer"
  }
  mock_outputs_merge_strategy_with_state = "shallow"
}

dependency "vpc_lattice" {
  config_path = "../../../platform/network/vpc-lattice"
  mock_outputs = {
    resource_gateway_eni_id = "eni-mock-lattice-rgw"
  }
  mock_outputs_merge_strategy_with_state = "shallow"
}

inputs = {
  resource_gateway_eni_id = dependency.vpc_lattice.outputs.resource_gateway_eni_id
  rds_writer_eni_id       = dependency.rds.outputs.rds_writer_eni_id
}

This is the network equivalent of a unit test. The output tells me:

✅ path_found = true → Network path is confirmed, all good
❌ path_found = false + explanations → Exactly which component is blocking traffic

And because it's a Terraform resource, if someone changes a security group rule or modifies a NACL, the next terragrunt apply will re-run the analysis and catch the regression. Automated network regression testing. That's NetDevOps.

🔄 Terragrunt Orchestration

The entire stack deploys with dependency resolution:

# Deploy everything in dependency order
export TF_WORKSPACE=dev
terragrunt run --all apply

Terragrunt resolves this graph automatically:

foundation/network/vpc
        ↓
foundation/network/security-groups
        ↓
platform/data/rds  ←→  platform/network/vpc-lattice
        ↓
observability/network/reachability-analyzer

And environment-specific values live in tfvars files:

# environments/dev/foundations.tfvars
vpc_cidr           = "10.100.0.0/16"
availability_zones = ["us-east-1a", "us-east-1b"]
database_subnets   = ["10.100.21.0/24", "10.100.22.0/24"]

# environments/dev/platform.tfvars
engine_version         = "16.6"
instance_class         = "db.t4g.medium"
instances = {
  writer = { instance_class = "db.t4g.medium" }
}
service_network_name    = "netdevops-external-dev"
resource_gateway_name   = "netdevops-rds-rgw-dev"
resource_config_name    = "netdevops-rds-config-dev"
custom_domain_name      = "db.dev.internal.netdevops.com"
ram_share_name          = "netdevops-rds-share-dev"
ram_external_principals = ["123456789012"]  # Network Account ID

# environments/prd/platform.tfvars (for comparison)
instance_class = "db.r6g.xlarge"
instances = {
  writer = { instance_class = "db.r6g.xlarge" }
  reader = { instance_class = "db.r6g.xlarge" }
}

Same code, different values. Dev → QA → Prod. That's the beauty of this approach.

So finally, I have the results:

Here we can find the service network and resource configuration association, pay attention to the Shareable option.

For other hand the reach ability analyzer succeeded.

And the RAM interface with the resource shared with network external account.

The Security Model

I want to call out the defense-in-depth approach because it's not accidental:

Layer	What It Does
VPC	No internet gateway, database-only subnets, zero public exposure
Security Groups	SG-to-SG references, only port 5432, minimum privilege
VPC Lattice	Auth type configurable (NONE for now, IAM for production)
RAM	Explicit principal sharing, requires acceptance from consumer
Encryption	Storage encrypted at rest, TLS in transit
Reachability Analyzer	Proves the path works — and only the intended path

No single layer is the "security layer." They all contribute. If any one fails, the others still protect you.

How the Consumer Connects

From the consumer's perspective (the Workload Account), it's beautifully simple:

The Network Account accepts the RAM share from the External Account
The Network Account reshares the Service Network with the Workload Account
The Workload Account associates their VPC to the shared Service Network
Connect to the database using the custom private DNS name — standard PostgreSQL client

App (Private subnet) → VPC Lattice (custom DNS: db.internal.example.com) → Resource Gateway → RDS

No routes to manage. No IPs to hardcode. No peering to maintain. The application connects to db.internal.example.com and doesn't even know it's crossing account boundaries. If you later migrate the database or change the underlying infrastructure, the consumer's connection string stays the same.

What I Learned

Building this out reinforced a few things for me:

1. VPC Lattice Resource Configurations are a game-changer. ARN-based targeting means you don't need to think about IPs, ports, or DNS records. You point to what you want to share, and Lattice figures out the rest.

2. Reachability Analyzer as IaC is underrated. I haven't seen many teams doing this, but it should be standard practice. If you can test your application code, you should test your network paths.

3. Layered architecture pays off. Yes, it's more files to manage. But when something breaks at 2 AM, you know exactly which layer to look at and which dependencies might be affected.

4. RAM sharing is cleaner than I expected. No network-level coupling between VPCs. The consumer doesn't need to know our CIDR ranges, our subnet layout, or our internal architecture. They just connect.

5. Security groups > CIDRs. Always. Security group references are the "zero trust" of VPC networking — they express intent, not implementation.

Cost and the Path to a "Networkless" World

This approach is significantly cheaper than Transit Gateway at scale.

Let me break it down. With Transit Gateway, you pay:

$0.05 per GB of data processed
$0.05 per hour per VPC attachment
Those costs multiply with every cross-account connection

With VPC Lattice, the model is different — you pay per request and per GB processed through the service network, but there's no per-attachment hourly cost. For database workloads with steady-state connections and moderate throughput, the savings are real.

But the bigger insight is architectural. VPC Lattice represents a shift toward what I call the networkless world — a modernization path where:

You stop thinking about routes and start thinking about resources
You stop managing IP connectivity and start managing access policies
You stop building network plumbing and start declaring what talks to what

This is the same evolution we saw from physical servers → VMs → containers → serverless. Now it's happening to networking: Transit Gateway (classic hub-and-spoke) → VPC Lattice (service mesh with resource-level targeting).

For organizations still running Transit Gateway architectures, VPC Lattice Resource Configurations offer a gradual migration path. You don't need to rip-and-replace — you can introduce Lattice for new cross-account patterns while TGW continues serving existing flows. Over time, as more services move to Lattice, the TGW footprint (and cost) shrinks naturally.

🔮 What's Next: The Deep Traffic Inspection Challenge

I'm already working on Part 2, which will cover the consumer side — accepting the RAM share and connecting from ECS tasks and Lambda, plus IAM authentication for zero-trust at the application layer.

But there's a bigger question I want to explore — and it's a hard one:

How do you implement deep traffic inspection with VPC Lattice?

In many financial organizations, regulatory compliance requires that all east-west traffic passes through a Network Firewall or third-party inspection appliance. Think PCI-DSS, SOX, or local financial regulations that mandate packet-level inspection for database access.

With Transit Gateway, the pattern is well-established: route traffic through an inspection VPC with AWS Network Firewall or a Palo Alto/Fortinet appliance. But with VPC Lattice, the traffic path is abstracted — you don't control the routing in the same way.

So the challenge for Part 3 becomes:

🎯 How do you satisfy deep packet inspection requirements in a VPC Lattice architecture? Can you combine Lattice with Network Firewall? Do you need a hybrid approach where Lattice handles service-to-service but TGW remains for inspected flows? Or does VPC Lattice's auth + encryption model satisfy the compliance intent without inspection?

This is the frontier of NetDevOps in regulated industries. And honestly, I don't think there's a single right answer yet — it depends on the specific regulatory framework and the organization's risk appetite.

If you're working in financial services or healthcare and have solved this, I'd really love to hear your approach in the comments.

If you want to dig into the code, the full project is structured as a reusable scaffold that you can adapt to your own cross-account patterns.

👇👇👇👇

velez94 / tofu-aws-netdevops-rds-external

Template project to show how to use vpc lattice and cross account access with RAM and Databases in AWS

NetDevOps External - AWS Infrastructure with VPC Lattice

Cross-account network connectivity infrastructure using AWS VPC Lattice, Aurora PostgreSQL, and Reachability Analyzer — managed with OpenTofu and Terragrunt.

Architecture

This project implements a Network Account (hub) pattern for cross-account connectivity. An external AWS account shares its service via RAM to this Network Account, which then reshares it to internal workload accounts using VPC Lattice — no VPC peering or Transit Gateway required.

┌─────────────────────────────────────────────────────────────────┐
│                   External AWS Account (Provider)                │
│                                                                 │
│  Aurora PostgreSQL ──► RAM Share (Service Network) ─────────┐   │
└─────────────────────────────────────────────────────────────┼───┘
                                                              │
┌─────────────────────────────────────────────────────────────▼───┐
│              This Account — Network Account (Hub)                │
│                                                                 │
│  VPC Lattice Service Network (received via RAM)                 │
│       │                                                         │
│       ├── Resource Gateway ──► Resource Configuration           │
│       │         (connects to external Aurora on port 5432)      │
│       │                                                         │
│       └── RAM Re-share ──► Workload Accounts                    │
│                                                                 │
│  Reachability Analyzer validates

…

View on GitHub

📚 Resources

If you made it this far — thanks for reading! 🙌 I'd love to hear if you're using VPC Lattice in production, or if you have other patterns for cross-account connectivity. Drop a comment below!

Happy building! 🚀

The opinions expressed in this post are my own and do not necessarily reflect those of my employer or AWS.

✨ Alejandro Velez, Platform Engineering Latam Lead @ GFT | AWS Ambassador

Automating AWS Well-Architected Reviews with Kiro CLI

Alejandro Velez — Mon, 01 Jun 2026 15:03:46 +0000

Level 300

As cloud architects and consultants, we've all been there: conducting Well-Architected Framework Reviews (WAFR) that take weeks of manual effort—collecting data, interviewing stakeholders, cross-referencing AWS documentation, and producing reports that often arrive too late to drive meaningful change. The community has responded with great tools: WA-Gen-AI analyzes IaC templates against best practices in minutes, the Well-Architected IaC Analyzer brings that to production scale, and the WA GenAI Workshop teaches prompt engineering patterns for reviews.

But after leading multiple WAFR engagements for financial services clients at GFT, I noticed a gap these tools don't cover: what happens when what clients claim in the WA Tool doesn't match what's actually running in their accounts? And how do you scale that verification across 6 pillars, multiple clients, and regulatory frameworks like DORA—without rebuilding the wheel every time?

The answer came from combining specialized AI agents with Kiro CLI's native subagent orchestration, automated gap analysis that cross-references claims with live account data, and a reusable scaffold that any consultant can clone and configure in minutes.

You're invited to explore how I built this approach—standing on the shoulders of existing tools while adding the verification and multi-client acceleration layer that was missing.

The Problem: Manual Assessments Don't Scale

Traditional WAFR assessments follow a predictable pattern:

Schedule workshops with multiple teams (2-3 weeks of calendar time)
Walk through 50+ questions per pillar manually
Trust that answers reflect reality (spoiler: they often don't)
Produce a report that's outdated by the time it's delivered
Repeat everything for the next client from scratch

Yes, the WA Tool supports custom lenses—you can upload your own compliance frameworks (DORA, SOX, internal standards) and evaluate workloads against them. This is powerful for structuring the questions. But the fundamental issue remains: the answers are self-reported. Custom lenses tell you what to ask, not whether the answers are true.

For regulated industries like banking (DORA, SOX compliance), this gap between stated practices and actual implementation goes undetected until an audit finds them—regardless of how well-crafted your custom lens is.

The Ecosystem: Tools That Accelerate WAFR Today

Before building this scaffold, I evaluated the existing tools. Each solves a piece of the puzzle:

Tool	What It Does	Strength
WA-Gen-AI / IaC Analyzer	Analyzes CloudFormation/Terraform templates against WA best practices using Bedrock	Answers ~50% of WA questions from IaC alone. Showcased at re:Invent 2024 with Commonwealth Bank
AWS Well-Architected Tool	Native console for structured reviews—questionnaires, milestones, improvement plans	Source of truth for formal WA reviews, custom lens support, Trusted Advisor integration
WA GenAI Workshop	AWS workshop on applying GenAI to accelerate reviews	Prompt engineering patterns, Bedrock integration

These are excellent—and complementary to what we're building. Here's the gap:

Capability	WA-Gen-AI / IaC Analyzer	WA Tool	This Scaffold
IaC template analysis	✅	❌	✅ (via MCP)
Structured questionnaire	❌	✅	✅ (agent prompts)
Live account verification	❌	❌	✅
Gap analysis (claims vs. reality)	❌	❌	✅
Trusted Advisor correlation	❌	✅ (native)	✅ (three-way validation)
Multi-client isolation	❌	Per-workload	✅
Parallel pillar execution	❌	Manual	✅ (subagents)
Compliance frameworks (DORA/SOX)	❌	Custom lenses	✅ (built-in)
Reusable scaffold/template	❌	❌	✅
Consultant workflow	❌	❌	✅

WA-Gen-AI analyzes what you intend to deploy (IaC). Our scaffold verifies what's actually running and compares it with what you claimed in the WA Tool. Together, they cover the full spectrum—design-time analysis AND runtime verification.

The ideal combined workflow:

WA-Gen-AI → Analyze IaC templates before deployment (shift-left)
WA Tool → Conduct formal review with structured questionnaires and custom lenses
This Scaffold → Verify claims against live accounts, correlate with Trusted Advisor, run parallel pillar assessments, produce evidence-based reports

The Solution: WAFR Assessment Scaffold

I created a scaffold project that provides the complete structure, AI agent configurations, and automation tooling to conduct comprehensive WAFR assessments. Think of it as a "quick start" for any new assessment engagement—clone it, configure it for your client, and let the agents do the heavy lifting.

Architecture Overview

well-architected-assessment-scaffold/
├── .kiro/
│   ├── agents/          # 7 specialized AI agents (one per pillar + general)
│   ├── prompts/         # Deep knowledge prompts for each assessment domain
│   └── steering/        # Project context (product, tech, structure)
├── clients/
│   └── {client-name}/   # Isolated client environments
│       ├── inputs/      # Client documentation & architecture
│       ├── data/        # Automated collection results
│       ├── config/      # Client-specific settings
│       └── outputs/     # Generated reports & dashboards
├── tools/
│   └── kiro-extensions/          # Utilities and helpers
├── config/
│   └── assessment/
│       └── pillars.yaml          # Pillar weights & priorities
└── docs/                          # Methodology, compliance, guidelines

The key design principle is client isolation with shared intelligence—every client gets their own data space, but all assessments leverage the same expert agents and proven methodology.

Prerequisites: AWS CLI Profile Configuration

Before running any assessment, you need a configured AWS CLI profile with read-only access to the target account. There are two ways to tell agents which profile to use:

Option 1: Specify in the prompt (recommended)

The simplest approach — tell the agent which profile to use when you invoke it:

kiro-cli chat --agent aws-wafr-expert "Conduct a full WAFR assessment for acme-corp. \
  Use AWS profile 'acme-corp-readonly' for all AWS operations. \
  Region: us-east-1."

The aws tool accepts a profile_name parameter on every call, so the agent will use it for all AWS CLI interactions throughout the session.

Option 2: Set in MCP server environment

For MCP servers that connect directly to AWS (CloudWatch, IAM, etc.), set the profile in the agent config:

"env": {
  "AWS_PROFILE": "acme-corp-readonly",
  "AWS_REGION": "us-east-1"
}

Client profile documentation

The scaffold provides a location to document each client's AWS access:

clients/acme-corp/config/aws/
└── profiles.ini          # Client AWS profile name, role ARN, region

Minimum IAM permissions required

The assessment role needs read-only access:

ReadOnlyAccess managed policy (broad coverage)
support:DescribeTrustedAdvisor* (Trusted Advisor — requires Business Support)
wellarchitected:Get*, wellarchitected:List* (WA Tool API)
ce:GetCost*, ce:GetDimensionValues, ce:GetTags (Cost Explorer)

Security note: Never use admin credentials for assessments. Create a dedicated read-only role. The agents only need to read account state, not modify it.

Hands On

Let's get into the code. 👽

Setting Up the Scaffold

velez94 / aws_well_architected_assement_scaffold

Well Architected Assement Scaffold Project with AI and Human on the loop

Well-Architected Assessment Scaffold

This is a scaffold/template project for AWS Well-Architected Framework Reviews (WAFR). It contains the essential Kiro AI configuration and a sample client structure to help you quickly start new assessment projects.

What's Included

`.kiro/` - Kiro AI Configuration

Contains all the AI agent configurations and prompts for conducting Well-Architected assessments:

agents/ - Specialized AI agents for each pillar:
- aws-wafr-expert.json - General WAFR expert
- aws-cost-exp.json - Cost Optimization pillar
- aws-security-expert.json - Security pillar
- aws-reliability-expert.json - Reliability pillar
- aws-performance-expert.json - Performance Efficiency pillar
- aws-opex-expert.json - Operational Excellence pillar
- aws-sustainability-expert.json - Sustainability pillar
prompts/ - Review prompts for each pillar:
- general-wafr-review.md
- cost-optimization-review.md
- security-pillar-review.md
- reliability-pillar-review.md
- performance-pillar-review.md
- operational-excellence-review.md
- sustainability-pillar-review.md
steering/ - Project guidance documents:
- product.md - Product requirements
- structure.md - Project structure guidelines
- tech.md - Technical specifications

`config/` - Project Configuration

assessment/ - Assessment configuration files:
- pillars.yaml - Well-Architected pillar definitions

`docs/` - Documentation

Comprehensive documentation for the assessment framework:

wafr/ -…

View on GitHub

Start by cloning and configuring for your client:

git clone https://github.com/velez94/aws_well_architected_assement_scaffold.git my-client-assessment
cd my-client-assessment
mv clients/acme-retail clients/acme-corp

The Agent Architecture

Each WAFR pillar has a dedicated Kiro CLI agent with specialized knowledge and MCP integrations. Here's the security expert configuration:

{
  "name": "aws-security-expert",
  "description": "AWS WAFR Security Pillar Expert",
  "prompt": "file://../prompts/security-pillar-review.md",
  "tools": ["read", "write", "shell", "aws"],
  "mcpServers": {
    "aws-documentation": {
      "command": "uvx",
      "args": ["awslabs.aws-documentation-mcp-server@latest"]
    },
    "iam": {
      "command": "uvx",
      "args": ["awslabs.iam-mcp-server@latest"]
    },
    "well-architected-security": {
      "command": "uvx",
      "args": ["awslabs.well-architected-security-mcp-server@latest"]
    },
    "cloudwatch": {
      "command": "uvx",
      "args": ["awslabs.cloudwatch-mcp-server@latest"]
    }
  }
}

Each agent connects to specific AWS MCP servers that give it real-time access to documentation, IAM analysis, security findings, and monitoring data—no hallucinations, just facts.

The full agent roster:

Agent	Pillar	Weight	MCP Integrations
`aws-security-expert`	Security	25%	IAM, WA Security, CloudWatch, Docs
`aws-reliability-expert`	Reliability	20%	AWS Support (Trusted Advisor), CloudWatch, Docs
`aws-cost-exp`	Cost Optimization	20%	Billing & Cost Management, Docs
`aws-performance-expert`	Performance	15%	CloudWatch, Docs
`aws-opex-expert`	Operational Excellence	15%	CloudTrail, CloudWatch, Docs
`aws-sustainability-expert`	Sustainability	5%	Billing & Cost Management, CloudWatch, Docs
`aws-wafr-expert`	General/Orchestration	—	All of the above + AWS Support, CloudTrail

Native Orchestration with Kiro Subagents

Here's where Kiro CLI shines. The built-in subagent system provides DAG-based pipeline execution with parallel stages, dependencies, and consolidated results. Each pillar agent runs as an independent stage, and a final consolidation stage merges everything:

# All 6 pillar assessments run IN PARALLEL, then consolidate
kiro-cli chat --agent aws-wafr-expert "Conduct a full WAFR assessment for acme-corp across all pillars. Use the client inputs in clients/acme-corp/inputs/ and cross-reference with actual AWS account findings. Produce a consolidated report with gap analysis, risk ratings, and prioritized remediation roadmap."

Under the hood, Kiro's subagent pipeline coordinates this as a DAG:

Stage	Agent	Execution
Security assessment	`aws-security-expert`	Parallel
Reliability assessment	`aws-reliability-expert`	Parallel
Cost assessment	`aws-cost-exp`	Parallel
Performance assessment	`aws-performance-expert`	Parallel
Operational Excellence	`aws-opex-expert`	Parallel
Sustainability	`aws-sustainability-expert`	Parallel
Consolidation	`aws-wafr-expert`	After all above

Why this matters: Traditional approaches require you to build and maintain orchestration code, handle failures, manage state, and coordinate outputs. With Kiro subagents, the infrastructure is the orchestrator. You focus on the assessment logic in your prompts and agent configurations, not on plumbing.

How Subagents Get Triggered

You don't need to manually define the pipeline in your prompt. The aws-wafr-expert agent has access to the subagent tool and understands from its prompt (the 4-phase methodology) that multi-pillar assessments should be delegated. When you give it a task like "assess all 6 pillars," it internally constructs the DAG:

{
  "task": "Full WAFR assessment for acme-corp",
  "stages": [
    {"name": "security", "role": "aws-security-expert", 
     "prompt_template": "Conduct security pillar assessment for {task}"},
    {"name": "reliability", "role": "aws-reliability-expert", 
     "prompt_template": "Conduct reliability pillar assessment for {task}"},
    {"name": "cost", "role": "aws-cost-exp", 
     "prompt_template": "Conduct cost optimization assessment for {task}"},
    {"name": "consolidation", "role": "aws-wafr-expert",
     "prompt_template": "Consolidate all findings into unified report for {task}",
     "depends_on": ["security", "reliability", "cost"]}
  ]
}

Each subagent:

Loads its own agent config (own MCP servers, own tools, own prompt)
Runs in an isolated session (no cross-contamination between pillars)
Reports results back via the summary tool when done
Can be monitored live with Ctrl+G in the TUI

You can also be explicit: "Use subagents to run security, reliability, and cost in parallel, then consolidate." But in most cases, the agent figures it out from the task scope and its methodology prompt.

Important: The orchestrating agent (aws-wafr-expert) must have "subagent" in its tools array — as documented here. Subagents cannot spawn sub-subagents — only the parent agent can orchestrate. Each subagent loads its own MCP servers independently, so pillar agents don't inherit the parent's connections.

To avoid approval prompts for each pillar agent spawn, configure trustedAgents in the orchestrator:

{
  "tools": ["read", "write", "shell", "aws", "subagent"],
  "toolsSettings": {
    "subagent": {
      "availableAgents": ["aws-security-expert", "aws-reliability-expert", 
        "aws-performance-expert", "aws-cost-exp", "aws-opex-expert", 
        "aws-sustainability-expert"],
      "trustedAgents": ["aws-*"]
    }
  }
}

This allows all aws-* pillar agents to be spawned without user confirmation — essential for unattended parallel execution.

The 4-Phase Assessment Flow

The assessment follows 4 logical phases, expressed through agent interactions:

Phase 1 - Data Collection: The aws-wafr-expert agent extracts WA Tool workload data, queries Trusted Advisor (when Business/Enterprise Support is available), and triggers automated account discovery via MCP integrations (Cost Explorer, CloudWatch, IAM).

Phase 2 - Gap Analysis: The same agent performs a three-way validation—comparing WA Tool responses vs. Trusted Advisor detections vs. actual account state. Discrepancies become the highest-priority inputs for pillar agents.

Phase 3 - Pillar Assessments: All 6 specialized agents run in parallel, each receiving the gap analysis context and performing deep-dive evaluation of their domain.

Phase 4 - Consolidation: The aws-wafr-expert merges all pillar results into a unified report with prioritized recommendations, risk ratings, and remediation roadmaps.

Pillar Configuration with Weights

The pillars.yaml defines assessment priorities—critical for financial services where security and reliability outweigh other concerns:

pillars:
  security:
    weight: 25
    priority: high
    focus_areas:
      - identity_access_management
      - data_protection
      - infrastructure_protection
      - detective_controls
      - incident_response

  reliability:
    weight: 20
    priority: high
    focus_areas:
      - foundations
      - workload_architecture
      - change_management
      - failure_management

  cost:
    weight: 20
    priority: high
    focus_areas:
      - practice_cloud_financial_management
      - expenditure_awareness
      - cost_effective_resources

Customize these weights per industry. Banking needs Security at 25%+; a startup might weight Cost Optimization higher. The agents use these weights to prioritize findings in the consolidated report.

Running a Pillar Assessment with Kiro CLI

Once your client inputs are in place, invoke a specialized agent:

# Run security pillar assessment
kiro-cli chat --agent aws-security-expert "Conduct a comprehensive security   assessment for acme-corp. Analyze the inputs in clients/acme-corp/inputs/security/  and cross-reference with actual AWS account findings."

# Run cost optimization review
kiro-cli chat --agent aws-cost-exp "Perform cost optimization analysis for 
  acme-corp workloads. Compare WA Tool responses with actual Cost Explorer data and Trusted Advisor cost findings."

The agents produce structured findings with evidence, risk ratings, and specific remediation steps—ready for executive or technical audiences.

Multi-Client Scalability

The scaffold supports multiple concurrent assessments with complete isolation:

clients/
├── acme-corp/        # Banking client (DORA + SOX)
│   ├── inputs/
│   ├── data/
│   └── outputs/
├── healthco/         # Healthcare client (HIPAA)
│   ├── inputs/
│   ├── data/
│   └── outputs/
└── retailx/          # Retail client (PCI-DSS)
    ├── inputs/
    ├── data/
    └── outputs/

Each client inherits the same expert agents and methodology but maintains completely separate data, configurations, and deliverables. This is critical for consulting firms managing multiple engagements—no cross-contamination, clean audit trails.

Integrating with the WA Tool API: Start Fresh or Continue an Existing Review

A key feature of this scaffold is its direct integration with the AWS Well-Architected Tool API. You can either start a new assessment from scratch or pull an existing review to contrast and continue the work:

# Option 1: Start from an existing workload assessment
kiro-cli chat --agent aws-wafr-expert "Pull workload assessment data from WA Tool workload ID abc12345-def6-7890-ghij-klmnopqrstuv. Extract all lens reviews,  answers, and milestones. Use this as baseline for gap analysis against the live account."

# Option 2: Start a fresh assessment (no prior WA Tool data)
kiro-cli chat --agent aws-wafr-expert "Create a new WAFR assessment for acme-corp.  No prior WA Tool review exists. Perform full account discovery and evaluate  against all 6 pillars from scratch."

The aws-wafr-expert agent uses the WA Tool API through its AWS tool integration to:

API Operation	Purpose
`ListWorkloads`	Discover existing workload assessments in the account
`GetWorkload`	Retrieve workload metadata and configuration
`GetLensReview`	Pull pillar-level review summaries and risk counts
`ListAnswers`	Extract all question responses for a given lens/pillar
`ListMilestones`	Track assessment history and improvement over time
`ListLensReviewImprovements`	Get AWS-recommended improvement items

When a prior review exists, the agent uses those answers as the "claims" side of the gap analysis—then verifies each claim against live account data and Trusted Advisor findings. This is where the real value emerges: you're not starting from zero, you're validating and extending work that was already done.

Two workflows, one scaffold:

This means the scaffold fits into any point of the assessment lifecycle—whether the client has been doing WA reviews for years or is starting their first one today.

Leveraging Trusted Advisor Integration

When the client has Business or Enterprise Support, the WA Tool automatically integrates with AWS Trusted Advisor—surfacing real-time checks across cost optimization, performance, security, fault tolerance, and service limits directly into the review. This is a powerful data source that many teams underutilize.

Our scaffold takes full advantage of this:

Trusted Advisor Category	How Kiro Agents Use It
Cost Optimization	`aws-cost-exp` cross-references TA findings (idle resources, underutilized instances) with Cost Explorer data for validated savings estimates
Security	`aws-security-expert` correlates TA security checks (open ports, IAM issues, MFA gaps) with Security Hub findings for comprehensive posture assessment
Fault Tolerance	`aws-reliability-expert` uses TA availability checks (multi-AZ, backup coverage, RDS redundancy) to validate reliability claims
Performance	`aws-performance-expert` leverages TA performance checks (overutilized instances, CloudFront optimization) alongside CloudWatch metrics
Service Limits	All agents flag service quota risks that could impact scalability or availability

With Business Support, Trusted Advisor provides full check access—over 100 checks that become automated evidence for the gap analysis. Without it, you're limited to core checks. This is why we recommend clients activate Business Support before starting the assessment.

The aws-wafr-expert agent queries Trusted Advisor findings via the AWS Support API and feeds them as context to each pillar agent. Combined with the WA Tool's native TA integration, this creates a three-way validation: what the client claimed (WA Tool answers) vs. what Trusted Advisor detected vs. what the account actually shows (live resource analysis via MCP).

The Gap Analysis: Claims vs. Reality

This is the differentiator. Traditional WAFR assessments trust the answers. Our approach verifies them.

Example findings from a real engagement:

WA Tool Claim	Trusted Advisor	Actual Finding	Risk
"MFA enabled for all users"	⚠️ IAM users without MFA detected	3 IAM users without MFA, root account MFA not hardware-based	HIGH
"Encryption at rest for all data"	— (not covered by TA)	2 S3 buckets with default encryption disabled, 1 RDS instance unencrypted	HIGH
"Automated backups configured"	⚠️ RDS backup retention < 7 days	Backup retention set to 1 day on production databases	MEDIUM
"Cost monitoring in place"	⚠️ No budget alerts	No budget alerts configured, no anomaly detection	MEDIUM

This evidence-based approach transforms the assessment from an opinion exercise into a factual audit—exactly what regulated industries need for DORA and SOX compliance.

Compliance Integration

The scaffold includes built-in support for regulatory frameworks commonly required in financial services, healthcare, and enterprise environments:

docs/compliance/
├── dora/     # Digital Operational Resilience Act (EU financial services)
├── hipaa/    # Health Insurance Portability (US healthcare)
└── sox/      # Sarbanes-Oxley (US financial reporting)

The security agent prompt includes specific DORA focus areas:

Financial Services Security (DORA Compliance): ICT risk management, operational resilience, third-party risk, Zero-Trust Architecture, Multi-Account Security

This means every security assessment automatically evaluates DORA readiness—no separate engagement needed. The compliance documentation provides the agents with regulatory context so their recommendations map directly to compliance requirements.

GenAI Patterns Applied

This scaffold implements several established GenAI patterns that ensure quality, safety, and reliability in an enterprise context. Understanding these patterns is important for anyone building AI-assisted workflows in regulated environments:

Human-in-the-Loop (HITL)

The most critical pattern here. AI agents collect data, analyze gaps, and generate recommendations—but a human architect validates and approves before anything reaches the client. This is non-negotiable for regulated industries.

Where HITL applies:

Review agent findings before consolidation into the final report
Validate risk ratings—the agent may flag something as HIGH that context makes MEDIUM
Approve remediation recommendations—ensure feasibility given client constraints
Sign-off on executive summaries—tone, framing, and business impact require human judgment

Retrieval-Augmented Generation (RAG)

Agents don't rely on training data alone. Through MCP integrations, they retrieve real-time data from:

AWS documentation (always current)
Live account state (IAM, CloudWatch, Cost Explorer)
WA Tool API (actual assessment responses)
Trusted Advisor (automated checks with Business/Enterprise Support)

This eliminates hallucinations about the client's actual environment.

Agentic Orchestration (Fan-out / Fan-in)

The subagent DAG implements a classic fan-out/fan-in pattern:

Fan-out: 6 specialized agents execute in parallel, each with domain expertise
Fan-in: The consolidation agent merges results into a unified deliverable

This reduces wall-clock time from sequential (6× pillar duration) to parallel (1× longest pillar duration).

Tool-Augmented Generation

Agents don't just generate text—they execute tools to gather evidence:

aws CLI calls for account interrogation and Trusted Advisor queries
MCP servers for structured data retrieval
File system reads for client documentation and architecture inputs

Every recommendation is backed by data the agent actually retrieved, not inferred.

Supervised Autonomy

The operating model is supervised autonomy:

Autonomous: Data collection, TA queries, resource inventory, metric gathering
Supervised: Gap analysis interpretation, risk prioritization, client-facing recommendations
Human-only: Final report approval, client presentation, remediation prioritization with stakeholders

This isn't fully autonomous AI replacing the architect. It's AI handling the 80% of toil (data collection, cross-referencing, formatting) so the architect focuses on the 20% that requires judgment, context, and client relationships.

Skills: Deep Knowledge On-Demand

One challenge with AI agents is context window management. If you load every possible best practice, compliance requirement, and remediation pattern into the agent prompt, you burn through context before the assessment even starts. But if you keep prompts lean, the agent lacks depth when it encounters specific scenarios.

Kiro CLI solves this with Skills—resources whose metadata (name + description) is loaded at startup, but whose full content is only pulled on-demand when the agent determines it's relevant.

How It Works

A skill is a Markdown file with YAML frontmatter:

---
name: dora-compliance-requirements
description: DORA regulatory requirements for ICT risk management, 
  operational resilience, and third-party risk. Use when assessing 
  financial services workloads for DORA compliance.
---

# DORA Compliance Requirements
## Key Articles for WAFR Assessment
### Article 5-16: ICT Risk Management
- Establish and maintain resilient ICT systems...
(detailed content only loaded when needed)

Referenced in agent config via skill:// URI:

{
  "resources": [
    "file://README.md",
    "skill://.kiro/skills/**/SKILL.md"
  ]
}

Skills vs. Prompts — When to Use Each

	Agent Prompt	Skill
Loaded	Always — full content in context	On-demand — only metadata at start
Purpose	Defines who the agent is (methodology, output format)	Provides deep reference knowledge for specific scenarios
Analogy	The architect's expertise	The reference books pulled from the shelf when needed
Size	Keep lean (< 5K tokens)	Can be large (2-20K tokens of detailed guidance)

The Scaffold's Skill Library

.kiro/skills/
├── compliance/
│   └── dora/SKILL.md              # DORA articles, AWS mapping, risk criteria
├── pillars/
│   ├── security/
│   │   └── iam-best-practices/SKILL.md    # IAM checks, common findings, evidence commands
│   ├── cost/
│   │   └── rightsizing-guide/SKILL.md     # Rightsizing analysis, savings estimation
│   └── reliability/
│       └── disaster-recovery/SKILL.md     # DR strategies by RTO/RPO, assessment checklist
└── remediation/
    └── quick-wins/SKILL.md                # 0-30 day low-effort high-impact actions

The security agent sees: "dora-compliance-requirements — Use when assessing financial services workloads". When it encounters a banking client, it pulls the full DORA skill with article-by-article AWS mappings. For a retail client, it never loads it—saving context for what matters.

Each pillar agent references only the skills relevant to its domain:

// aws-security-expert.json
"resources": [
  "file://README.md",
  "skill://.kiro/skills/pillars/security/**/SKILL.md",
  "skill://.kiro/skills/compliance/**/SKILL.md",
  "skill://.kiro/skills/remediation/**/SKILL.md"
]

This means you can grow the skill library indefinitely—adding new compliance frameworks, service-specific patterns, industry guidance—without impacting agent performance. The agents stay fast and focused, pulling depth only when the assessment requires it.

Knowledge Bases: Searching Through Client Data

Skills solve the "deep reference knowledge" problem. But what about the client's own documentation—architecture diagrams, runbooks, security policies, infrastructure configs? These can be 50+ files that are too large to load into context but need to be searchable during the assessment.

Kiro CLI's Knowledge Base feature provides persistent semantic search across sessions. You index client inputs once, and agents can search through them on-demand using natural language queries.

How It Works

Define a knowledge base in the agent config with auto-sync:

{
  "resources": [
    "file://README.md",
    "skill://.kiro/skills/**/SKILL.md",
    {
      "type": "knowledgeBase",
      "source": "file://./clients",
      "name": "Client Inputs & Architecture",
      "indexType": "best",
      "include": ["**/*.md", "**/*.yaml", "**/*.json", "**/*.txt"],
      "exclude": ["**/outputs/**", "**/draft/**"],
      "autoUpdate": true
    }
  ]
}

When the agent starts, all client documentation gets indexed semantically. The agent then searches it during the assessment—pulling only the relevant snippets into context when needed.

The Three-Layer Context Strategy

Layer	Mechanism	When Loaded	Best For
Prompts	`file://../prompts/`	Always (full content)	Agent identity, methodology, output format
Skills	`skill://.kiro/skills/`	On-demand (by description match)	Deep reference knowledge, compliance frameworks
Knowledge Bases	`knowledgeBase` resource	Searchable (semantic query)	Large client documentation, architecture inputs

Prompts tell the agent how to assess. Skills give it what to look for. Knowledge Bases let it find relevant client context without loading everything into memory.

This layered approach means an agent can handle a client with 100+ input documents without ever hitting context limits—it searches, retrieves the relevant 2-3 snippets, and uses them to produce evidence-based findings.

Best Practices and Lessons Learned

After multiple engagements using this scaffold, here are the patterns that consistently deliver the best results:

Start with steering files: Clear product, tech, and structure documentation in .kiro/steering/ gives agents the context they need to produce relevant findings instead of generic advice. Invest time here—it pays dividends across every assessment.
Limit MCP tools per agent (~50 max): Each tool definition consumes ~400 tokens of context window. At 100+ tools, you're burning 40K+ tokens before any work begins. The aws-wafr-expert orchestrator needs only 4 servers (~26 tools) for coordination—leave the heavy toolsets (IAM 29 tools, Billing 33 tools, CloudWatch 19 tools) to the specialized pillar agents that actually use them.
Gap analysis is non-negotiable: The comparison between WA Tool answers and actual account state consistently reveals the highest-value findings. Never skip this phase—it's where the scaffold earns its keep.
Pillar weights matter: Customize pillars.yaml per industry. Banking needs Security at 25%+; a startup might weight Cost Optimization higher. These weights drive prioritization in the consolidated report.
Client isolation is a feature, not overhead: Each client assessment gets its own scaffold project—clone the scaffold, configure it, and you have a completely independent repository with its own git history, access controls, and lifecycle. No shared state, no cross-contamination. For consulting firms, this means clean handoffs, simple access management per engagement, and compliance with data handling requirements out of the box.
Activate Business Support first: Recommend clients enable Business or Enterprise Support before the assessment. The full Trusted Advisor check access (100+ checks) dramatically improves evidence quality.
Use Kiro CLI for parallel assessments: While one agent analyzes security, another can simultaneously evaluate cost optimization. The subagent system coordinates everything—leverage it.
Keep inputs clean and structured: The quality of agent output directly correlates with the quality of client documentation in inputs/. Garbage in, garbage out still applies—even with AI.
Supervised execution for critical findings: Let agents run autonomously for data collection, but review and validate high-risk recommendations before presenting to clients. The human-in-the-loop pattern is your quality gate.
Iterate on prompts: The prompts in .kiro/prompts/ are living documents. After each engagement, refine them based on what worked and what produced noise. This is how the scaffold gets smarter over time.

The scaffold is open for the community to use, extend, and improve. Whether you're a solo consultant or part of a large practice, the goal is the same: deliver better assessments, faster, with evidence.

Thank you for your time and support. Please remember to follow us for additional updates.

✨ Alejandro Velez, Platform Engineering Latam Lead @ GFT | AWS Ambassador

References:

Here my new post using Kiro and MCP to improve and modernize my current workflow for Authorization pipeline using Identity Center and IAM Access Analyzer

Alejandro Velez — Sun, 08 Mar 2026 22:59:16 +0000

Re-imagine DevSecOps with AWS - CD applied to Authorization with IAM Identity Center and AWS IAM Access Analyzer

Alejandro Velez for AWS Community Builders ・ Mar 7

#security #aws #devsecops #genia

Re-imagine DevSecOps with AWS - CD applied to Authorization with IAM Identity Center and AWS IAM Access Analyzer

Alejandro Velez — Sat, 07 Mar 2026 02:37:42 +0000

Level 300

Before diving into the technical depths of permission sets and CDK pipelines, let’s take a brief journey through the story so far. Originally, our exploration began with the challenge of managing access in the cloud task that’s both critical and complex as organizations scale. In the first installment, we introduced the foundations: using AWS IAM Identity Center as the central hub for managing user identities and permissions efficiently across multiple AWS accounts. We looked at how a consistent and automated approach helps streamline authorization, reduce manual errors, and improve auditability.

The next chapter followed with practical examples illustrating how to create and deploy permission sets via Infrastructure-as-Code. Here, we learned to define reusable templates for permissions, enabling teams to replicate best practices and accelerate onboarding for new projects. Along the way, pitfalls emerged—like misconfigured policies or improper use of wildcards—which provided valuable lessons on the importance of policy validation and iterative testing in a DevSecOps environment.

Reviewing recent news, in June 2025 AWS introduced internal access findings, which spot principals in an AWS organization with access to key resources like S3 buckets, DynamoDB tables, and RDS snapshots by analyzing various policy types. Results appear on a unified dashboard for fast response or automated notifications via Amazon EventBridge, improving compliance and data security.

In February 2026, AWS added multi‑Region replication for Identity Center, allowing workforce identities and permission sets to sync across regions. This boosts resiliency, supports data residency requirements, and ensures consistent access control for global deployments.

Additionally, the integration of generative AI solutions into authorization workflows is reshaping how policies are analyzed and managed. With the introduction of AWS Kiro —an AI-powered assistant for access analysis—teams can now automatically detect policy anomalies, receive context-aware recommendations, and accelerate remediation efforts. This part will explore these innovations and demonstrate how they can be practically applied to build secure, efficient, and adaptive authorization pipelines in the modern enterprise.

The Challenge

As an AWS security specialist, you face the task of modernizing your organization's authorization management workflow. The goal is to enhance the provisioning and management of access controls across AWS environments while preserving robust multi-region replication capabilities, upholding DevSecOps best practices, and ensuring that human oversight remains integral to the process. At the same time, you must increase resiliency, proactively mitigate risks, and introduce AI-driven solutions to streamline operations and strengthen security posture.

How can AI enhance this workflow? 🧐

Integrating AI into authorization workflows—especially with tools like Kiro— dramatically enhances both efficiency and security. AI-powered assistants can automatically analyze complex IAM policies for anomalies or risky configurations, such as improper use of wildcards or misconfigured permissions, which are often the sources of failed pipeline executions and compliance gaps. By leveraging generative AI, organizations receive real-time, context-aware recommendations for policy corrections, reducing manual troubleshooting and accelerating remediation. AI solutions can also automate policy reviews, flag potential issues before deployment, and offer suggestions to align with best practices, thus minimizing human error and strengthening security posture. In addition, AI-driven analytics help teams continuously monitor and adapt policies to evolving threats, ensuring adaptive and resilient access control across global environments. Altogether, AI capabilities transform the workflow by making policy management more proactive, scalable, and responsive to both technical and regulatory demands.

Let's start by enhancing our workflow from traditional CI/CD to incorporating an** Architect in the Loop approach**.

In the Architect in the Loop (AITL) model, architects collaborate with AI systems, which generate options and analyze trade-offs. The architect reviews, contextualizes, and approves output based on technical and organizational needs. Here, AI performs most tasks but consults the architect for decisions and guidance when necessary.

The diagram depicts a multi account AWS environment with a centralized management approach. It illustrates how various services, identity providers, and automation tools interact to enforce security, governance, and continuous deployment across multiple AWS accounts.

Key Components

Accounts and Regions
• AWS Account – DevSecOps: This is the primary development and security operations account where IAM Idenity Center is delegate.
• AWS Account – Master: Acts as the central governance hub, managing permissions and policies across the organization for itself.
• AWS Account – Security Tooling Account: Hosts security focused tools and services for monitoring and analysis, managing permissions and policies across the organization for itself.
Core Services
• Key Management Service (KMS): Provides encryption and key management for sensitive data.
• CloudWatch: Monitors system and application metrics.
• Secrets Manager: Stores and rotates secrets securely.
• AWS Chatbot: Enables chat based monitoring and alerts.
• Lambda: Runs serverless functions for custom automation.
• IAM Access Analyzer: Analyzes policies for unintended access.
• S3 Reports Bucket: Stores logs and reports for auditing.
Identity and Access Management
• Identity Center and Third Party IdP icons are shown in the diagram, indicating that identity federation and centralized access management are part of the architecture.
• IAM Access Analyzer is highlighted, showing that policy analysis is performed to ensure least privilege access using validate-aws-policies python package.
CI/CD Pipeline
The CI/CD pipeline is visualized as a sequence of steps:
• CodeCommit/GitHub (source code repository)
• CodeBuild (build step, Synth,cdk pipelines self-mutate, management environment, Validate PermissionsSet))
• Manual Approval (human gate for critical deployments)
Infrastructure Deployment
• CloudFormation templates are used for both Prepare and Deploy stages, indicating that the infrastructure is provisioned through IaC.
• The Permissions Set step shows that IAM permission sets are defined and applied as part of the deployment process.
Model Context Protocol (MCP) Integration
• Kiro IDE – Agents is highlighted, indicating that an MCP compatible IDE is used to generate policies and manage custom agents.
• The AWS MCP Server provides the necessary endpoint for agents to interact with AWS services.
• The label mcproxy-for-aws@latest refers to a proxy that facilitates secure communication between agents and AWS APIs, ensuring that policy generation and SOPs (Standard Operating Procedures) are executed safely.

Workflow Summary

Source Control – Code is stored in CodeCommit/GitHub.
Build – CodeBuild compiles the code and prepares the environment.
Validation – Permissions sets are validated to ensure compliance.
Manual Approval – A human reviewer authorizes the deployment.
Deployment – CloudFormation templates are used to provision resources.
Policy Generation – The Kiro IDE and AWS MCP Server generate policies based on requirements, leveraging the mcproxy for secure API calls.

Security and Governance Highlights

• Least privilege access is enforced through IAM Access Analyzer and Permissions Set validation.
• Centralized governance is maintained via the Master account and delegated account, which oversees permissions and policies across all accounts.
• Continuous monitoring is provided by CloudWatch and Secrets Manager, ensuring operational visibility and secure secret handling.

Hands On

It is time to create!

The first step is improving the security Engineer experience like builder or operator so enable the use of smart IDE like Kiro and custom agent for context.

Here is the agent specification:

{
  "name": "secops",
  "description": "Specialized agent for managing AWS IAM Identity Center CDK project with permission sets, policies, and pipeline automation",
  "prompt": "You are an expert in AWS IAM Identity Center, AWS CDK (Python), and CI/CD pipelines. Your expertise includes:\n\n- IAM Identity Center permission sets and assignments\n- IAM Access Analyzer policy validation\n- AWS CDK infrastructure as code (Python)\n- CodePipeline and CodeBuild automation\n- Amazon Q Developer chatbot integrations (Slack/Teams)\n- Policy validation and security best practices\n- GitHub Actions and CodeCommit PR validation\n\nYou help with:\n- Creating and managing permission sets\n- Validating IAM policies with Access Analyzer\n- Troubleshooting CDK deployments\n- Configuring pipeline notifications\n- Setting up PR validation workflows\n- Implementing least privilege access\n\nAlways prioritize security, follow AWS best practices, and provide minimal, focused implementations.",
  "tools": [
    "fs_read",
    "fs_write",
    "execute_bash",
    "grep",
    "glob",
    "code",
    "use_aws",
    "aws___search_documentation",
    "aws___read_documentation",
    "web_search"
  ],
  "allowedTools": [
    "fs_read",
    "grep",
    "glob",
    "code",
    "aws___search_documentation"
  ],
  "toolsSettings": {
    "fs_write": {
      "allowedPaths": [
        "src/**",
        "project_configs/**",
        "tests/**",
        ".github/**",
        "scripts/**",
        "docs/**",
        ".kiro/**",
        "*.md",
        "*.yaml",
        "*.yml"
      ],
      "deniedPaths": [
        "cdk.out/**",
        ".venv/**",
        "node_modules/**",
        "__pycache__/**"
      ]
    },
    "execute_bash": {
      "allowedCommands": [
        "cdk synth",
        "cdk diff*",
        "cdk deploy*",
        "cdk ls",
        "pip install*",
        "python -m pytest*",
        "validate-aws-policies*",
        "git status",
        "git diff",
        "git log*",
        "aws iam*",
        "aws sso*",
        "aws identitystore*",
        "aws codepipeline*",
        "aws logs*",
        "aws s3 ls*"
      ],
      "autoAllowReadonly": true
    },
    "use_aws": {
      "allowedServices": [
        "iam",
        "sso",
        "sso-admin",
        "identitystore",
        "codepipeline",
        "codebuild",
        "s3",
        "logs",
        "cloudformation",
        "accessanalyzer",
        "secretsmanager",
        "chatbot"
      ],
      "autoAllowReadonly": true
    }
  },
  "resources": [
    "file://README.md",
    "file://project_configs/**/*.yaml",
    "file://cdk.json",
    "file://requirements.txt",
    "skill://.kiro/skills/**/SKILL.md"
  ],
  "hooks": {
    "agentSpawn": [
      {
        "command": "echo '📋 IAM Identity Center Project Context:' && echo '  - CDK Version:' $(cdk --version 2>/dev/null || echo 'Not installed') && echo '  - Python:' $(python --version 2>&1) && echo '  - AWS Profile:' ${AWS_PROFILE:-'default'} && echo '  - Project:' $(basename $(pwd))",
        "description": "Show project environment info"
      },
      {
        "command": "if [ -d 'project_configs/policies' ]; then echo '🔍 Validating IAM policies...' && validate-aws-policies -d project_configs/policies -c --quiet || echo '⚠️  Policy validation failed'; else echo 'ℹ️  No policies directory found'; fi",
        "description": "Auto-validate IAM policies with Access Analyzer"
      }
    ]
  },
  "mcpServers": {
    "aws-mcp": {
      "command": "uvx",
      "args": [
        "mcp-proxy-for-aws@latest",
        "https://aws-mcp.us-east-1.api.aws/mcp",
        "--metadata",
        "AWS_REGION=us-east-2"
      ],
      "timeout": 100000
    }
  },
  "keyboardShortcut": "ctrl+shift+i",
  "welcomeMessage": "🔐 IAM Identity Center Manager ready! I can help with:\n  • Permission sets and policy validation\n  • CDK deployments and troubleshooting\n  • Pipeline notifications (Slack/Teams)\n  • PR validation workflows\n  • Access Analyzer integration\n\nWhat would you like to work on?"
}

The key point here is the prompt for a specialized agent designed to manage AWS IAM Identity Center projects using the AWS CDK (Cloud Development Kit) with a focus on permission sets, policies, and automation for CI/CD pipelines. This agent is described as an expert in several areas:}

• AWS IAM Identity Center: Managing permission sets and assignments, which define what users and groups can access within AWS accounts.
• AWS CDK (Python): Using infrastructure-as-code techniques to automate the creation and management of AWS resources.
• CI/CD pipelines: Automating deployment workflows with CodePipeline and CodeBuild, ensuring reliable and repeatable processes.
• Policy validation: Leveraging IAM Access Analyzer to check and validate security policies, helping maintain least-privilege access and compliance.

Now the agent skills. Agent Skills

Skills are portable instruction packages that follow the open Agent Skills standard. They bundle instructions, scripts, and templates into reusable packages that Kiro can activate when relevant to your task.
Kiro supports the Agent Skills standard, so you can import skills from the community or other compatible AI tools and share your own skills across the ecosystem.

Here is an example of my skill:

---
name: iam-identity-center-project
description: "Complete guide for IAM Identity Center CDK project including permission sets, policy validation, pipeline automation, and chatbot notifications. Use when working with this project's architecture, deployment, or troubleshooting."
keywords: [iam, identity-center, permission-sets, cdk, pipeline, chatbot, policy-validation]
---

# IAM Identity Center CDK Project

## Project Overview

This CDK project manages AWS IAM Identity Center permission sets with automated policy validation, CI/CD pipelines, and chatbot notifications.

## Key Components

### 1. Permission Sets Management
- **Location**: `project_configs/environment_options/*.yaml`
- **Stack**: `src/cdkv2_manage_identity_center_stack.py`
- Define permission sets with inline policies and managed policies
- Support for principal_name (auto-resolved) or principal_id (direct)

### 2. Policy Validation
- **Tool**: `validate-aws-policies` (IAM Access Analyzer)
- **Pipeline Step**: ValidatePermissionSet in CodeBuild
- **Formats**: JSON, HTML, Markdown reports
- **Storage**: S3 bucket (labvel-secure-reports)

### 3. CI/CD Pipeline
- **Stack**: `src/cdkv2_manage_identity_center_pipeline.py`
- **Stages**: Source → Build → UpdatePipeline → ValidatePermissionSet → Deploy
- **PR Validation**: GitHub Actions + CodeCommit triggers
- **Notifications**: Amazon Q Chatbot (Slack/Teams)

### 4. Chatbot Integration
- **Location**: `src/lib/chatbot/amazon_q_chatbot.py`
- **Features**: Native CodePipeline notifications + report summaries via webhook
- **Platforms**: Slack and Microsoft Teams
- **Reports**: Lambda posts validation summaries to chat

## Source Code Structure

src/
├── cdkv2_manage_identity_center_stack.py    # Main stack - permission sets
├── cdkv2_manage_identity_center_pipeline.py # Pipeline stack
└── lib/
    ├── chatbot/
    │   ├── amazon_q_chatbot.py              # Chatbot construct
    │   └── lambda/notification_enricher.py  # Lambda for reports
    ├── sns/sns_codestart_notifications.py   # SNS notifications
    └── custom_resources/                    # Custom resources

**Note**: Use `fs_read`, `code`, or `grep` tools to explore source files on-demand.

## Quick Commands
...

With this approach the security specialist can create friendly and secure policies and check before uploading

The results

Pipeline execution:

The slack custom notification and amazon Q message

And the reports:

Conclusion

Leverage AWS's latest features—such as multi-region Identity Center replication and advanced IAM Access Analyzer tools—and integrate generative AI assistants like Kiro. This approach combines automated policy analysis, real-time anomaly detection, and context-aware remediation with architect-in-the-loop oversight, allowing AI to handle routine tasks while security professionals review and approve critical changes. The result is a resilient, adaptive, and scalable workflow that proactively reduces risk, enforces compliance, and empowers teams to respond rapidly to evolving security and regulatory demands.

For more please contact us!

✨ Alejandro Velez, Platform Engineering Latam Lead @ GFT | AWS Ambassador

Agentic AI Development with Kiro: From Zero to SaaS Platform

Alejandro Velez — Thu, 20 Nov 2025 19:30:23 +0000

Level 300

Recently, I came up with an idea inspired by my personal experience. I am a professional calisthenics athlete and frequently organize events where a recurring challenge is providing users with a seamless way to track their results. As event managers, registering and publishing results, managing competition timing, scoring systems, and different modalities are also critical concerns. On the other hand, as an AWS solutions architect and Devsecops expert, my first thought was how I could combine both worlds to create a solution that is fast, secure, and modern—without sacrificing best practices.

You're invited to explore how agentic AI development with Kiro can transform event management and SaaS platform creation. Whether you're an AWS developer, DevOps or DevSecOps engineer, SaaS architect, AI practitioner, platform engineer, technical product owner, or event manager, join us as we delve into modern cloud and AI technologies designed to streamline event management and result tracking. Discover best practices in secure, scalable, cloud-native SaaS solutions, and accelerate your development with serverless architecture and innovative AI workflows.

Prototyping with Kiro: Accelerating Development Through Serverless Best Practices

In my pursuit of optimizing both the software development lifecycle (SDLC) and platform engineering, I have been actively exploring a variety of AI workflows and practices. The goal is to enhance efficiency and innovation within the development process. However, the immediate need is to create a rapid prototype that supports my concept effectively while adhering to best practices.

To achieve this, I prioritized serverless architectures for their speed, scalability, and cost-effectiveness, ensuring that the solution remains secure and can grow alongside user demand. Among the available options, I selected Kiro as the primary tool for development. Kiro stands out by bringing structure to AI coding through spec-driven development, enabling me to produce robust prototypes swiftly while maintaining high standards in both security and scalability.

Some key features are:

Natural prompts to structured requirements
Architectural designs backed by best practices
Discrete tasks that map to requirements

Laying the Foundation for a Reliable Prototype

Creating a reliable prototype is inherently complex, especially given the possibility that the idea may gain traction and eventually move into production. In such scenarios, it is essential to minimize technical debt from the outset, ensuring that the prototype is prepared for a seamless transition to production. Therefore, the process of selecting appropriate technologies and establishing solid foundational practices becomes a critical aspect of moving from prototyping to a Minimum Viable Product (MVP).

Technical Stack and Development Practices

For the front-end, I selected a React web application to ensure a responsive and interactive user experience. The application's architecture adheres to the Twelve-Factor App methodology, which provides a framework for building scalable and maintainable software-as-a-service solutions.
To maintain high standards of reliability and security, I strictly followed AWS best practices throughout the development process. The system is designed using event-driven architecture, allowing for efficient handling of asynchronous tasks and improved scalability.

Infrastructure as Code (IaC) is implemented using the AWS Cloud Development Kit (CDK), streamlining the process of defining and managing cloud resources in a repeatable and version-controlled manner.

For continuous integration and deployment, I integrated GitHub with self-hosted runners and AWS CodeBuild. This setup enables a seamless and automated flow from code committees to deployment, supporting rapid iteration and robust DevOps workflows.

For testing purposes, I am utilizing Nova Act to facilitate the creation of UI tests with Playwright; however, further details on this topic will be covered in subsequent posts.

I use both Kiro IDE and Kiro CLI to parallelize the project task, while the IDE works in spec tasks the CLI helps me to create some backend components for infrastructure, debugging and fixing errors.

Hands On

At the beginning of the development, I created a CDK scaffold project and inserted some files with basic project rules and deep knowledge about the system:

First according to https://kiro.dev/docs/getting-started/first-project/ we create the project and setup the steering files:

Steering files give Kiro project context, so it understands your codebase, conventions, and needs. Start by selecting Generate Steering Docs in the Kiro pane. Kiro then creates steering documents in .kiro/steering/ that outline your product, tech stack, project structure, and conventions.

Some examples for steering data files:

product.md

Athleon Platform

Multi-tenant calisthenics competition management platform with role-based access control (RBAC).

## Core Functionality

- **Organizations**: Multi-tenant teams with owner/admin/member roles
- **Events**: Competition lifecycle management (draft → published → completed)
- **Athletes**: Profile management, event registration, score submission
- **Scoring**: Advanced calculation engine with multiple scoring systems
- **WODs**: Workout templates (event-scoped and global)
- **Categories**: Competition divisions (event-scoped and transversal)
- **Scheduling**: Tournament brackets and session management

## User Roles
...

tech.md

# Tech Stack

## Infrastructure

- **IaC**: AWS CDK (TypeScript)
- **Compute**: AWS Lambda (Node.js 18.x)
- **Database**: DynamoDB (single-table per domain)
- **API**: API Gateway REST API with Cognito authorizer
- **Auth**: Amazon Cognito User Pools
- **Storage**: S3 (event images)
- **Events**: EventBridge (domain event bus + central bus)
- **Deployment**: CDK deploy with esbuild bundling

## Backend

- **Runtime**: Node.js 18.x
- **SDK**: AWS SDK v3 (@aws-sdk/client-*)
- **Testing**: Jest, Vitest, fast-check (property-based testing)
- **Shared Layer**: Lambda layer at `layers/athleon-shared/nodejs`

## Frontend

- **Framework**: React 18 + Vite
- **Routing**: React Router v6
- **State**: Zustand + React Query (@tanstack/react-query)
- **Auth**: AWS Amplify v6
- **UI**: Custom components with @aws-amplify/ui-react
- **i18n**: i18next + react-i18next
- **Testing**: Vitest + React Testing Library
- **E2E**: Playwright (in e2e-tests/)

## Common Commands

### Infrastructure

...

Now, build a feature with specs:
Specs transform high-level feature ideas into detailed implementation plans through three phases:

Requirements- User stories with acceptance criteria in EARS notation
Design- Technical architecture and implementation approach
Tasks- Discrete, trackable implementation steps

For example, one feature was: Create the RBAC system to keep the roles and responsibilities for different platform users (Super Admin, athletes, organizers and spectators). This prompt generates a structure and consolidate workflow from requirements to the code:

requierements.md

# Requirements Document

## Introduction

This specification defines a role-based user onboarding system for the Athleon platform using a single Amazon Cognito User Pool. The system must support three distinct user roles (athlete, organizer, and super admin) with different access levels and onboarding workflows. Athletes and organizers can self-register through the frontend with role selection, while super admins are created exclusively through backend scripts.

## Glossary

- **Cognito User Pool**: AWS managed user directory service that handles user authentication and stores user attributes
- **Pre-signup Trigger**: AWS Lambda function invoked by Cognito before user registration is completed
- **Custom Attribute**: User-defined attribute stored in Cognito user profile (e.g., custom:role)
- **Athlete**: A user who competes in events and views leaderboards
- **Organizer**: A user who creates and manages events, categories, and WODs
- **Super Admin**: A platform administrator with full system access
- **Self-registration**: User-initiated signup process through the frontend application
- **RBAC**: Role-Based Access Control system that determines user permissions based on assigned roles
- **JWT Token**: JSON Web Token containing user claims including the custom:role attribute
- **Onboarding Flow**: The complete process from initial signup to authenticated access

design.md

## Architecture

### High-Level Components

┌─────────────────┐
│  Frontend App   │
│  (React + AWS   │
│   Amplify UI)   │
└────────┬────────┘
         │
         │ 1. Signup with role selection
         ▼
┌─────────────────────────────────────────┐
│     Amazon Cognito User Pool            │
│  ┌───────────────────────────────────┐  │
│  │  Pre-signup Lambda Trigger        │  │
│  │  - Validates role selection       │  │
│  │  - Sets custom:role attribute     │  │
│  │  - Rejects super_admin attempts   │  │
│  └───────────────────────────────────┘  │
│  ┌───────────────────────────────────┐  │
│  │  Pre-token Generation Trigger     │  │
│  │  - Adds role to JWT claims        │  │
│  └───────────────────────────────────┘  │
└─────────────────┬───────────────────────┘
                  │
                  │ 2. JWT with role claim
                  ▼
┌─────────────────────────────────────────┐
│     API Gateway + Lambda Functions      │
│  - Extract role from JWT token          │
│  - Enforce role-based permissions       │
│  - Route to appropriate services        │
└─────────────────────────────────────────┘
...

tasks.md

# Implementation Plan

- [x] 1. Update Pre-Signup Lambda Trigger
  - Modify `lambda/auth/pre-signup-trigger.js` to extract and validate role from signup request
  - Implement logic to accept 'athlete' or 'organizer' roles
  - Implement logic to reject 'super_admin' role with error
  - Default to 'athlete' for missing or invalid roles
  - Add comprehensive CloudWatch logging for all role assignments
  - _Requirements: 1.2, 1.3, 1.4, 1.5, 5.1, 5.2, 5.3, 5.4, 5.5, 8.1, 8.2_

- [x] 1.1 Write property test for pre-signup trigger
  - **Property 1: Role assignment consistency**
  - **Property 2: Super admin rejection**
  - **Property 3: Default role assignment**
  - **Validates: Requirements 1.2, 1.3, 1.4, 1.5, 2.4, 5.1, 5.2, 5.3, 5.4**
...

Here we need to extend the capabilities using Model Context Protocol (MCP) and add more context:

• Access specialized knowledge bases and documentation
• Integrate with external APIs and services
• Use domain-specific tools and utilities
• Connect to databases and cloud services

I initially integrate two MCP servers:

{
  "mcpServers": {
    "awslabs.core-mcp-server": {
      "command": "uv",
      "args": [
        "tool",
        "run",
        "--from",
        "awslabs.core-mcp-server@latest",
        "awslabs.core-mcp-server.exe"
      ],
      "env": {
        "FASTMCP_LOG_LEVEL": "ERROR",
        "AWS_PROFILE": "labvel-dev",
        "AWS_REGION": "us-east-2",
        "aws-foundation": "true",
        "solutions-architect": "true"
      },
      "disabled": false,
      "disabledTools": [
        "aws_knowledge_aws___get_regional_availability",
        "aws_knowledge_aws___list_regions",
        "pricing_get_bedrock_patterns",
        "cost_explorer_get_cost_comparison_drivers"
      ]
    },
    "playwright": {
      "command": "npx",
      "args": [
        "@playwright/mcp@latest"
      ],
      "disabled": false
    }
  }
}

The Results

I'm excited to share the outcomes from our hands-on experience building agentic AI solutions with Kiro! 😁

The following section captures the valuable lessons I've learned, the best practices I've discovered, and how thoughtful choices have helped us grow from an initial idea to a polished SaaS platform for managing calisthenics events and athlete profiles.

The IDE Project setup

Here is the review of my project setup, as you can see there are MCP, steering project files and some specs.

The Complete system architecture for tournament auto-advance

Scheduling Domain

Watch here some exception to strict DDD and boundex contex domain patterns, in this scenario I choice make read data directly between context because use rest API or service integration increase the complexity for this use case.

Scoring domain Integration

The landing page

Best Practices and lessons learned

Make sure that your scaffold project and product rules information are clear and clean.
Runs in supervised way to avoid mistakes, Kiro makes many tasks correctly but sometimes I need reject some changes because violate some rules or use bad approach to solve a technical issue.
Don’t integrate more than 50 MCP tools, this consumes context and could be slow.
Use Kiro CLI for specific troubleshooting and parallel tasks that do not modify the current tasks or files while specs are running.
Create specialized Kiro CLI agents for each SDLC domain and product domains after creating protype or MVP.
Keep flexibility and tradeoff according to your use case. Sometimes apply deep practices like DDD add more complexity however hard guardrails and guidelines are security and operational excellent.
Keep the repository clean, while prompting and refining many md files are created, some related to upgrading summaries or analysis to debug, if those files are added as context to the agent could generate confusion and reprocess.

Thank you for your time and support. Please remember to follow us for additional updates.

✨ Alejandro Velez, Platform Engineering Latam Lead @ GFT | AWS Ambassador

Using Amazon Q and custom agents for IaC

Alejandro Velez — Wed, 17 Sep 2025 03:31:17 +0000

Alejandro Velez for AWS Community Builders

Sep 17 '25

Enhancing Infrastructure as Code Development and Operations with Amazon Q, MCP, and Thoth Framework

#devops #ai #aws #iac

5 min read

Enhancing Infrastructure as Code Development and Operations with Amazon Q, MCP, and Thoth Framework

Alejandro Velez — Wed, 17 Sep 2025 03:23:41 +0000

Level 300

With each phase of digital transformation, new approaches are introduced for developing and implementing solutions. Beginning with scripting tools such as Ansible and Chef, and progressing through innovations like Terraform, CDK, Pulumi, and today’s AI-driven agentic and autonomous systems, methodologies continually evolve. Some practices become obsolete, while fresh strategies emerge—challenging engineers to adapt, innovate, and drive progress in solution creation and maintenance.

It is common for DevOps professionals to upgrade Infrastructure as Code (IaC) regularly; maintaining clean infrastructure dependencies at scale can be challenging, but many processes are now automated. Thoth framework simplifies dependency management, automates template generation and integrates seamlessly with existing workflows, reducing manual effort and minimizing errors in large-scale infrastructure projects building and managing IaC templates created with tools such as Terraform or Tofu, leveraging wrappers like Terragrunt and Terramate.

Let me show you how common tasks can be automatically accelerated using traditional approaches and modern practices with custom agents like Amazon Q.

The left side: Development

Things are constantly evolving; tasks like coding are being redefined by tools from developer assistants to agentic AI, moving us closer to fully autonomous development. Soon, writing code may seem as outdated as using an abacus, but human interaction is important and necessary for critical thinking, architecture decisions, continuous improvement and alignment with business strategies. As a cloud architect, developer, or engineer, it is essential to define the infrastructure composition with careful consideration of application-driven design and operational models. Adhering to best practice consistent with the well-architected framework and internal guidelines—is necessary to ensure optimal performance and reliability.

So, How can I do this with minimal effort, time, and resources?

Companies use internal developer platforms with blueprints and quick starts to reduce toil, lower the learning curve, and enable self-service through established paths. Developers must interact using the correct interfaces. AI agents now serve as intuitive interfaces, exposing platform capabilities via MCP and allowing for tailored agents for each SDLC task.

Let’s begin with the code. 👽

Start by creating a custom agent with Amazon Q for IaC, including platform context via MCP and a custom CLI. This approach manages tasks like infrastructure composition, compliance, scanning, and reporting, while maintaining traditional practices such as git best practices.

The following picture depicts this setup.

The system interfaces directly with AWS services via the AWS SDK and leverages OpenTofu for infrastructure provisioning, ensuring consistent and reproducible deployments across multiple environments.

You can add any complementary MCP service from list but be careful verify the source:

awslabs / mcp

Official MCP Servers for AWS

Open source MCP servers for AWS

A suite of specialized MCP servers that help you get the most out of AWS, wherever you use MCP.

Open source MCP servers for AWS

View on GitHub

modelcontextprotocol / servers

Model Context Protocol Servers

Model Context Protocol servers

This repository is a collection of reference implementations for the Model Context Protocol (MCP), as well as references to community-built servers and additional resources.

Important

If you are looking for a list of MCP servers, you can browse published servers on the MCP Registry. The repository served by this README is dedicated to housing just the small number of reference servers maintained by the MCP steering group.

Warning

The servers in this repository are intended as reference implementations to demonstrate MCP features and SDK usage. They are meant to serve as educational examples for developers building their own MCP servers, not as production-ready solutions. Developers should evaluate their own security requirements and implement appropriate safeguards based on their specific threat model and use case.

The servers in this repository showcase the versatility and extensibility of MCP, demonstrating how it can be used to give Large…

View on GitHub

Hands On

Requirements

WSL or ubuntu 24.04
python >= 3.12
thothctl >= 0.5.3
opentofu >= 1.10.6
terragrunt >= 0.88.0

Preparing the local environment

Bootstrap you environment with the necessary tools following the next steps:

a. Download and install thothctl from pypi official repository.

pipx install thothcl

b. Install amazon Q agent and Amazon Q for your IDE, terragrunt, tofu, uv, and pipx and other tools running or just runs the devtocontainers environment.

thothctl init environment  #environment for interactive mode

Select the tools according to the recommended versions. If you already have the tools installed, please run.

thothctl check environment

Creating custom Amazon Q agent (thoth agent)

Please create the AWS Builder Id

a. Use thothctl to initialize the project with the scaffold template or clone the repository.

The Custom agent configuration files are stored as JSON files in specific directories:

Project-level custom agents .amazonq/cli-agents/{agent-name}.json
Available only within the specific project directory and its subdirectories.

The Amazon Q Developer CLI searches for a custom agent by following a defined order of precedence:
• Local custom agents first - Checks for custom agents in the current working directory
• Global custom agents second - Falls back to custom agents in your home directory
• Built-in default - Uses the default agent if no custom agent is found

ℹ️ Please visit for best practices and deep knowledge: 👉 Custom Agents Management 👈

For this scenario the scaffold project template looks like:


├── LICENSE
├── README.md
├── common
│   ├── common.hcl
│   ├── common.tfvars
│   └── variables.tf
├── docs
│   └── catalog
│       ├── catalog-info.yaml
│       ├── docs
│       │   ├── general
│       │   ├── guidelines
│       │   │   ├── architecture-definition.md
│       │   │   └── iac-composition-guidelines.md
│       │   ├── images
│       │   │   ├── DiagramArchitecture.png
│       │   │   └── graph.svg
│       │   └── index.md
│       └── mkdocs.yml
├── root.hcl
└── stacks
    ├── application
    │   ├── compute
    │   │   ├── alb
    │   │   │   ├── README.md
    │   │   │   └── terragrunt.hcl
    │   │   └── asg
    │   └── storage
    │       ├── efs
    │       └── s3
    ├── foundation
    │   ├── iam
    │   │   ├── policies
    │   │   └── roles
    │   │       └── terragrunt.hcl
    │   └── network
    │       ├── security-groups
    │       └── vpc
    │           ├── README.md
    │           └── terragrunt.hcl
    ├── observability
    │   └── monitoring
    │       ├── cloudwatch
    │       └── prometheus
    └── platform
        ├── containers
        │   ├── ecr
        │   ├── eks-control-plane
        │   │   └── terragrunt.hcl
        │   └── eks-nodegroups
        └── data
            ├── elasticache
            └── rds

You can find it in:

thothforge / terragrunt_project_scaffold

Scaffold for terragrun projects using thoth framework

Terragrunt Project Scaffold

A production-ready Terragrunt template for AWS infrastructure deployment with GitOps integration and best practices.

Overview

This scaffold provides a standardized project structure for managing AWS infrastructure using Terragrunt, with built-in support for:

Multi-environment deployments
Remote state management with S3 and DynamoDB
Code quality tools (TFLint, pre-commit hooks)
GitOps workflows
Modular architecture

Project Structure

#{project_name}#/
├── .thothcf.toml              # Template configuration
├── .gitignore                 # Git ignore rules
├── .tflint.hcl               # TFLint configuration
├── .pre-commit-config.yaml   # Pre-commit hooks
├── root.hcl                  # Root Terragrunt configuration
├── common/
│   ├── common.hcl            # Common variables and provider config
│   └── variables.tf          # Shared variable definitions
├── stacks/
│   ├── foundation/           # Core infrastructure layer
│   │   ├── network/vpc/      # VPC, subnets, routing
│   │   └── iam/roles/        # Service roles and policies
│   ├── platform/             # Shared services layer
│   │   └── containers/
│   │       └── eks-control-plane/  # EKS cluster
│   ├── application/          #

…

View on GitHub

So, the custom agent provides the minimum mcp servers, context and tools. Agents can be created based on environment, technology, specialty or for specific projects for example when using a monorepo structure to store both application and infrastructure code.

Here is the baseline agent setup:

{
  "name": "thoth",
  "description": "IaC and GitOps specialist THOTH agent for IaC deployments",
  "mcpServers": {
     "thothctl": {
      "command": "thothctl",
      "args": ["mcp", "server", "--stdio"]
    },

    "git": {
       "command": "uvx",
      "args": ["mcp-server-git"],
      "timeout": 30000
    },
    "terraform": {
      "command": "docker",
      "args": [
        "run",
        "-i",
        "--rm",
        "hashicorp/terraform-mcp-server"
      ]
    },
    "awslabs.aws-diagram-mcp-server": {
      "command": "uvx",
      "args": [
        "awslabs.aws-diagram-mcp-server"
      ],
      "env": {
        "FASTMCP_LOG_LEVEL": "ERROR"
      },
      "autoApprove": [],
      "disabled": false
    }
  },
  "tools": [
    "fs_read",
    "fs_write",
    "execute_bash",
    "use_aws",
    "@git",
    "@thothctl",
    "@terraform"
  ],
  "allowedTools": [
    "fs_read",
    "use_aws",
    "@git/git_status",
    "@git/git_log",
    "@git/git_diff"
  ],
  "toolAliases": {
    "@git/git_status": "status",
    "@git/git_log": "log",
    "@git/git_diff": "diff"
  },
  "toolsSettings": {

    "fs_write": {
      "allowedPaths": [
        "stacks/**",
        "common/**",
        "modules/**",
        "*.hcl",
        "*.tf",
        "*.tfvars",
        "*.yaml",
        "*.yml",
        "*.toml",
        "*.md",
        "docs/**"
      ]
    }

  },
  "resources": [
    "file://README.md",
    "file://LICENSE",
    "file://.thothcf.toml",
    "file://root.hcl",
    "file://common/common.hcl",
    "file://common/variables.tf",
    "file://.tflint.hcl",
    "file://.pre-commit-config.yaml",
    "file://.gitignore",
    "file://stacks/**/*.hcl",
    "file://stacks/**/*.tf",
    "file://docs/**/*.md"
  ]

}

The agent has the resources definition block:

In docs/catalog/docs/guidelines we include two guidelines, one for architecture definitions and other for IaC composition guidelines.

b. Finally, start a chat with the agent in project folder and create some stacks :


➜  q chat --agent thoth
✓ terraform loaded in 1.05 s
✓ git loaded in 1.81 s
✓ awslabs.aws-diagram-mcp-server loaded in 2.02 s
✓ thothctl loaded in 2.12 s

For example:

Thanks for reading and sharing! 🤠

The next blogs offer more examples and explains how traditional and agentic AI can be combined for optimal results. 🥸

✨ Alejandro Velez, Platform Engineering Latam Lead @ GFT | AWS Ambassador

Here the 3rd part of this series!

Alejandro Velez — Mon, 04 Aug 2025 01:48:07 +0000

Alejandro Velez for AWS Community Builders

Aug 3 '25

GitOps and IaC at Scale – ArgoCD and Open Tofu – Part 3 – Hardening and Manage users

#devops #aws #terraform #gitops

7 min read

GitOps and IaC at Scale – ArgoCD and Open Tofu – Part 3 – Hardening and Manage users

Alejandro Velez — Sun, 03 Aug 2025 23:40:39 +0000

Level 400

Earlier blogs covered scalable delivery, architecture patterns, and code samples. This section provides additional production-ready examples and best practices using the Well Architected Framework and GitOps with ArgoCD.

Architecture Overview

Now, suppose that you must apply some custom addons for the environment, also enable on-boarding application developer’s teams and platform teams to manage their projects and applications with the least manual effort, integration with AWS ecosystem like IAM identity Center and apply best practices for ArgoCD setup. For accomplishing this the following image depicts the high-level reference architecture with the main services:

Here the new resources are ALB as an ingress gateway for Argo server, Route 53 to enable a Hosted Zone for public Domain, AWS Certificate management for SSL trough the ALB, and IAM Identity Center for SSO capabilities.

For production environments there could be a private hosted Zone and use PKI architecture internal to enable SSL access.

For other hand, *how can you be modeling the groups and access to the Argo projects and applications? *
There is not correct answer and depending on your operational model (Conway’s Law) for example, the following image depicts a simple group for each team around a workload.

Assigning permissions should follow the principle of the least privilege, separating application and project actions. For example, the application engineering team can run all actions on Applications but only watch and update the platform and addon applications.

Hands On

It’s time to create code.

First, we must modify the hub blueprint to enable argo cd best practices
❌ Don’t use Default admin for management- Disable it.
❌ Don’t use the default project
❌ Don’t use local users just if it is completely necessary for CLI or external automatons.
✅ Create Roles and assign to groups instead of individual assignments.

Now, according to Argo cd documentation the SSO is configured with IAM Identity Center.

A working Single Sign-On configuration using Identity Center (AWS SSO) has been achieved using SAML (with Dex)

To manage IAM Identity Center in AWS, applications must be created because the CreateApplicationAPI only supports custom OAuth 2.0 apps. Third-party SAML or OAuth 2.0 apps must be set up via their respective services or the AWS console. Application creation cannot be automated with Terraform, but user assignments can still be handled through IaC.

Initially, a custom Terraform module was developed to establish the ArgoCD hardening baseline. This module leverages the gitops bridge module as its foundation and incorporates SSO configuration, along with group and default role setup and baseline specifications.

The module name is terraform-helm-hardening-gitops-bridge

You can find the source code module here 👉

thothforge / terraform-helm-hardening-gitops-bridge

GitOps bridge extended module with hardennig practices and examples

Terraform Hardening GitOps Bridge Module

A comprehensive Terraform module that provides a hardened GitOps bridge for Amazon EKS clusters, implementing security best practices and enterprise-grade configurations for GitOps workflows using ArgoCD.

📖 Table of Contents

Features

Prerequisites

Architecture

Quick Start

Examples

Configuration Options

Security Best Practices

Deployment Patterns

Monitoring and Observability

Performance Optimization

Advanced Security Configuration

DNS and Ingress Configuration

Customization Options

Cost Optimization

Best Practices

Operational Procedures

FAQ

Additional Resources

Contributing

License

Support

🚀 Features

Core Capabilities

GitOps Integration: Seamless integration with ArgoCD for GitOps workflows

Security Hardening: Enterprise-grade security configurations and best practices

Multi-Repository Support: Support for addons, platform, and workloads repositories

Flexible Deployment: Single cluster or hub-spoke architecture support

Comprehensive Addons: Pre-configured essential Kubernetes addons

Security Features

RBAC Integration: Role-based access control with customizable permissions

SSO Support: Microsoft Entra ID (Azure AD) integration

Certificate Management: Automated SSL/TLS certificate provisioning

…

View on GitHub

or 👉

registry.terraform.io

The main files are: bootstrap/argocd-values.yaml where the default values are created and parser some parameters from terraform parameters like default project name, dns or argo host, kind of ingress, tags, subnets and more.

# bootstrap/argocd-values.yaml
global:
  topologySpreadConstraints:
    - maxSkew: 1
      topologyKey: "topology.kubernetes.io/hostname"
      whenUnsatisfiable: "ScheduleAnyway"
    # Default logging options used by all components
  logging:
    # -- Set the global logging format. Either: `text` or `json`
    format: text
    # -- Set the global logging level. One of: `debug`, `info`, `warn` or `error`
    level: debug



configs:
  params:
    server.insecure: true
  # SSO configuration with Entra ID
  cm:

    url: https://${argo_host}
    dex.config: |
      logger:
        level: debug
        format: json
      connectors:
      - type: saml
        id: aws
        name: "AWS IAM Identity Center"
        config:
          # You need value of Identity Center APP SAML (IAM Identity Center sign-in URL)
          ssoURL: ${sso_assertion_url} #https://portal.sso.yourregion.amazonaws.com/saml/assertion/id
          # You need `caData` _OR_ `ca`, but not both.
          #<CA cert (IAM Identity Center Certificate of Identity Center APP SAML) passed through base64 encoding>
          caData: ${ca_data_iam_app} 
          # Path to mount the secret to the dex container
          entityIssuer: https://${argo_host}/api/dex/callback
          redirectURI: https://${argo_host}/api/dex/callback
          usernameAttr: email
          emailAttr: email
          groupsAttr: groups

  rbac:
    policy.csv: |
      p, role:platform-team, applications, *, */*, allow
      p, role:platform-team, projects, *, */*, allow
      p, role:platform-team, clusters, *, *, allow
      p, role:platform-team, repositories, *, *, allow
      p, role:platform-team, certificates, *, *, allow

      p, role:platform-team, logs, get, */*, allow
      p, role:platform-team, exec, create, */*, allow
      p, role:platform-team, applicationsets, *, */*, allow
      p, role:platform-team, accounts, get, */*, allow
      p, role:platform-team, sessions, create, */*, allow
      p, role:platform-team, sessions, delete, */*, allow
      p, role:platform-team, projects, get, *, allow
      p, role:platform-team, projects, list, *, allow
      p, role:platform-team, projects, update, *, allow
      p, role:platform-team, projects, create, *, allow

      p, role:platform-team, actions, *, */*, allow
      g, ${admin_idp_group_id}, role:platform-team

    policy.default: role:deny
    policy.matchMode: glob
    scopes: '[groups, email]'
server:
  autoscaling:
    minReplicas: 2
    maxReplicas: 20
    targetCPUUtilizationPercentage: 80
    targetMemoryUtilizationPercentage: 80
  serviceAccount:
    annotations:
      eks.amazonaws.com/role-arn: "${argocd_irsa_role_arn}"
  service:
    type:  "NodePort"
  ingress:
    enabled: "${enable_argo_ingress}"
    hostname: "${argo_host}"
    controller: "aws"
    ingressClassName: "alb"
    tls: true
    aws:
      backendProtocolVersion: GRPC
      serviceType: "NodePort" #"ClusterIP"
    annotations:
      alb.ingress.kubernetes.io/group.name: argocd
      alb.ingress.kubernetes.io/backend-protocol: HTTP
      alb.ingress.kubernetes.io/healthcheck-protocol: HTTP
      alb.ingress.kubernetes.io/listen-ports: '[{"HTTP":80},{"HTTPS":443}]'
      alb.ingress.kubernetes.io/scheme: "${aws_load_balancer_type}"
      alb.ingress.kubernetes.io/security-groups: "${argo_ingress_sg}"
      alb.ingress.kubernetes.io/certificate-arn: "${acm_certificate_arn}"
      alb.ingress.kubernetes.io/ssl-policy: ELBSecurityPolicy-TLS-1-2-2017-01
      alb.ingress.kubernetes.io/ssl-redirect: "443"
      alb.ingress.kubernetes.io/healthcheck-path: /healthz
      alb.ingress.kubernetes.io/subnets: "${ingress_subnets}"
      alb.ingress.kubernetes.io/tags: "${required_tags}"
    path: /

    # -- Ingress path type. One of `Exact`, `Prefix` or `ImplementationSpecific`
    pathType: Prefix
    #extraRules:
    #- http:
    #    paths:
    #    - path: /argocd
    #      pathType: Prefix
    #      backend:
    #        service:
    #          name: 'argo-cd-argocd-server-grpc'
    #          port:
    #            name: 'https'
  ingressGrpc:
    # -- Enable an ingress resource for the Argo CD server for dedicated [gRPC-ingress]
    enabled: false #"${enable_argo_ingress}"
    hostname: "${argo_host}"
    ingressClassName: "alb"
    annotations:
      alb.ingress.kubernetes.io/group.name: argocd
      alb.ingress.kubernetes.io/backend-protocol: HTTPS
      alb.ingress.kubernetes.io/backend-protocol-version: GRPC
      alb.ingress.kubernetes.io/listen-ports: '[{"HTTP":80},{"HTTPS":443}]'
      alb.ingress.kubernetes.io/healthcheck-protocol: HTTP
      alb.ingress.kubernetes.io/target-type: "ip"
      alb.ingress.kubernetes.io/ssl-redirect: "443"
      alb.ingress.kubernetes.io/healthcheck-path: /healthz
      alb.ingress.kubernetes.io/success-codes: "0-99"
      alb.ingress.kubernetes.io/security-groups: "${argo_ingress_sg}"
      alb.ingress.kubernetes.io/certificate-arn: "${acm_certificate_arn}"
      alb.ingress.kubernetes.io/ssl-policy: ELBSecurityPolicy-TLS-1-2-2017-01
      alb.ingress.kubernetes.io/subnets: "${ingress_subnets}"
      alb.ingress.kubernetes.io/tags: "${required_tags}"
controller:
  replicas: 3
  serviceAccount:
    annotations:
      eks.amazonaws.com/role-arn: "${argocd_irsa_role_arn}"
  metrics:
    enable: true
  priorityClassName: "system-node-critical"
  podAnnotations:
    prometheus.io/scrape: true
    prometheus.io/port: 8082
    prometheus.io/path: "/metrics"
  service:
    annotations:
      prometheus.io/scrape: true
      prometheus.io/port: 8082
      prometheus.io/path: "/metrics"


repoServer:
  replicas: 2
  serviceAccount:
    annotations:
      eks.amazonaws.com/role-arn: "${argocd_irsa_role_arn}"
    metrics:
      enable: true
    priorityClassName: "system-node-critical"
    podAnnotations:
      prometheus.io/scrape: true
      prometheus.io/port: 8084
      prometheus.io/path: "/metrics"
    service:
      annotations:
        prometheus.io/scrape: true
        prometheus.io/port: 8084
        prometheus.io/path: "/metrics"


notificationsController:
  enable: true

redis-ha:
  enabled: true
  waitForVotes: false
  quorum: 2

applicationSet:
  metrics:
    enable: true
  replicas: 2
  podAnnotations:
    prometheus.io/scrape: true
    prometheus.io/port: 8085
    prometheus.io/path: "/metrics"
  service:
    annotations:
      prometheus.io/scrape: true
      prometheus.io/port: 8085
      prometheus.io/path: "/metrics"

For other hand, the argo project default template is: bootstrap/default_argproj.yaml

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: ${default_argoproj_name}
  namespace: argocd
  labels:
    argocd.argoproj.io/instance: argoprojects-platform-team
spec:
  description: Team for TI to manage Argo projects and applications
  sourceRepos: ${jsonencode(repositories)}
  destinations:
    - namespace: '*'
      server: '*'
  clusterResourceWhitelist:
    - group: '*'
      kind: '*'
  namespaceResourceWhitelist:
    - group: '*'
      kind: '*'

You can modify according to the baseline’s requirements; this is the platform project like default.

The platform addons and platform appsets just receive the parameter for default project in argoCD but keep as the beginning of this series.

Other kinds of resources for the module are security groups and default secrets, if the SSO is not enabled for platform, for admin, developers and view only users.

For other hand, the module can create a certificate and CNAME in a public hosted zone. Is usual that these resources are managed from other team, but in this setup in the same account exists a delegate hosted zone for internal use. This allows use external DNS addon to enable easy integration and register additional services in your control plane.

Sensitive data like ca data and group id could be loaded from parameter store provided for security team. Or save in secrets manager to mount the secret onto the dex container in the argocd-dex-server Deployment.

The module can be utilized with Terragrunt as follows:

#eks_control_plane-terragrunt.hcl
include "root" {
  path   = find_in_parent_folders("root.hcl")
  expose = true
}

#include "kubectl_provider" {
#  path = find_in_parent_folders("/common/additional_providers/provider_kubectl.hcl")
#}

include "k8s_helm_provider" {
  path = find_in_parent_folders("/common/additional_providers/provider_k8s_helm.hcl")
}

dependency "eks" {
  config_path = "${get_parent_terragrunt_dir("root")}/infrastructure/containers/eks_control_plane"
  mock_outputs = {
    cluster_name                       = "dummy-cluster-name"
    cluster_endpoint                   = "dummy_cluster_endpoint"
    cluster_certificate_authority_data = "dummy_cluster_certificate_authority_data"
    cluster_version                    = "1.31"
    cluster_platform_version           = "1.31"
    oidc_provider_arn                  = "dummy_arn"
    node_security_group_id = "sg-xasr18923d"
    cluster_primary_security_group_id = "sg-xasr18923d"
  }
  mock_outputs_merge_strategy_with_state = "shallow"
}
dependency "eks_role" {
  config_path = "${get_parent_terragrunt_dir("root")}/infrastructure/iam/eks_role"
  mock_outputs = {
    iam_role_arn = "arn::..."
  }
  mock_outputs_merge_strategy_with_state = "shallow"

}
dependency "vpc" {
  config_path = "${get_parent_terragrunt_dir("root")}/infrastructure/network/vpc"
  mock_outputs = {
    vpc_id = "vpc-04e3e1e302f8c8f06"
    public_subnets = [
      "subnet-0e4c5aedfc2101502",
      "subnet-0d5061f70b69eda14",
    ]
    private_subnets = [
      "subnet-0e4c5aedfc2101502",
      "subnet-0d5061f70b69eda14",
      "subnet-0d5061f70b69eda15",
    ]
  }
  mock_outputs_merge_strategy_with_state = "shallow"
}


locals {
  # Define parameters for each workspace
  env = {
    default = {

      environment = "control-plane"
      default_project = include.root.locals.environment.locals.default_project
      enable_argo_ingress = true
      sso_assertion_url = "https://portal.sso.<SSO_REGION>.amazonaws.com/saml/assertion/<id>"
      ca_data= "BASE 64 CA Data"
      argo_host_dns = {
        domain_name            = "argocd.devsecops.labvel.io"
        zone_id                = "Z101221820UZWIPQ27722"
        aws_load_balancer_type = "internet-facing"
        validation = "public"
      }
      oss_addons = {
        enable_argo_workflows = true
        #enable_foo            = true
        # you can add any addon here, make sure to update the gitops repo with the corresponding application set
      }

      addons_metadata = merge(
        include.root.locals.environment.locals.addons_repos_properties
      )


        tags = {
          Environment = "control-plane"
          Layer       = "Containers"
        }
      }

    "dev" = {

      create = true
    }
    "prod" = {

      create = true
    }
  }
  # Merge parameters
  environment_vars = contains(keys(local.env), include.root.locals.environment.locals.workspace) ? include.root.locals.environment.locals.workspace : "default"
  workspace        = merge(local.env["default"], local.env[local.environment_vars])
}


terraform {
  source ="tfr:///thothforge/hardening-gitops-bridge/helm?version=1.0.1"

}

inputs = {
  cluster_name                       = dependency.eks.outputs.cluster_name
  cluster_endpoint                   = dependency.eks.outputs.cluster_endpoint
  cluster_platform_version           = dependency.eks.outputs.cluster_platform_version
  oidc_provider_arn                  = dependency.eks.outputs.oidc_provider_arn
  node_security_group = dependency.eks.outputs.cluster_primary_security_group_id
  vpc_id         = dependency.vpc.outputs.vpc_id
  private_subnet_ids = dependency.vpc.outputs.private_subnets
  public_subnet_ids = dependency.vpc.outputs.public_subnets

  cluster_certificate_authority_data = dependency.eks.outputs.cluster_certificate_authority_data
  enable_argo_ingress = local.workspace["enable_argo_ingress"]
  argo_host_dns = local.workspace["argo_host_dns"]
  #external_dns_domain_filters = [local.workspace["argo_host_dns"]]
  sso_assertion_url = local.workspace["sso_assertion_url"]
  ca_data= local.workspace["ca_data"]

  admin_idp_group_id ="318xx789-x071-70f5-63x6-xx21233xxx33"

  tags = local.workspace["tags"]

}

Finally you can login into argocd hub control plane using the url or through AWS SSO login interface.

The upcoming post will include additional customization options and add-ons.
Thanks for reading and sharing.🙌 👽 😃

GitOps and IaC at Scale – AWS, ArgoCD, Terragrunt, and OpenTofu – Part 2 – Creating Spoke environments

Alejandro Velez — Sun, 20 Apr 2025 00:18:12 +0000

Level 400

Hi clouders, in the previous blog you can explore the considerations to deploy at scale gitops and IaC, in this post you can learn more about how to deploy spokes cluster using the GitOps bridge framework and specific use cases with AWS enterprise capabilities and security best practices.

Architecture Overview

Spoke clusters and Scenarios

According to the best practices there are many considerations and scenarios, for example:

You can have one stack or IaC repository with the definitions for each account, team or squat. It depends on your internal organizations and share responsibility model. For this scenario suppose that the model is a decentralized DevOps Operational Model, and each squat has their custom IaC for each project. Consider keeping clear separation and isolation between environments and make match your platform strategy, for example, some organizations have a single hub for managing dev, QA and prod cluster, others have a hub to manage each environment cluster and other, and others have one hub for previous environments and another for production.
Another key point to note is the capacity planning and applications by team, some organizations allow share the same cluster for an organizational unit and keep the applications grouped by namespaces in each environment, others prefer to have one environment and cluster by workload or application. Considering the networking and security considerations has main pain and challenges for both scenarios. In this series the main assumption is that each workload has a dedicated cluster and account by environment but there is a transversal platform team that manages the cluster configuration and control plane. The following table describe the relationship between scenario and AWS accounts:

Scenario	AWS Account	Clusters
Single Hub – N Spokes	1 for HUB – N account by environment	1 cluster Hub, N Spoke Clusters by environment
M Hub - N spokes	M accounts By Hub environments – N account by environments	M Hub Clusters, N Environment Clusters

The Figure 1 depicts the architecture for this scenario.

Figure 1. GitOps bridge Deployment architecture in AWS

The FinOps practices are key point independent of your resources distribution you must consider what is the best strategy for track cost and shared resources.

Hands On

First, modify the hub infrastructure stacks to add the stack to manage the credentials cross account and allow the CI infrastructure agents to take them to register the cluster in the control plane. Also, create the role for Argocd to enable the authentication between Argocd hub and spoke cluster.

Updating Control Plane Infrastructure

So, let’s set up the credentials according to the best practices and the scenario described in the previous section, the cluster credentials are stored in parameter store and share with the organizational units for each team or business unit.

You must enable RAM as trusted service in your organization from Organizations management account and RAM Console.

For this task, a local module terraform-aws-parameter-store was created:

└── terraform-aws-ssm-parameter-sotre
    ├── README.md
    ├── data.tf
    ├── main.tf
    ├── outputs.tf
    └── variables.tf

The module creates the parameter and shares with organization id, organizational unit, or account id principals.

Now, using terragrunt the module is called to create a new stack or terragrunt unit.

#parameter_store-terragrunt.hcl

include "root" {
  path = find_in_parent_folders("root.hcl")
  expose = true
}

dependency "eks" {
  config_path = "${get_parent_terragrunt_dir("root")}/infrastructure/containers/eks_control_plane"
  mock_outputs = {
    cluster_name = "dummy-cluster-name"
    cluster_endpoint = "dummy_cluster_endpoint"
    cluster_certificate_authority_data = "dummy_cluster_certificate_authority_data"
    cluster_version = "1.31"
    cluster_platform_version = "1.31"
    oidc_provider_arn =  "dummy_arn"
    cluster_arn = "arn:aws:eks:us-east-2:105171185823:cluster/gitops-scale-dev-hub"
  }
  mock_outputs_merge_strategy_with_state = "shallow"
}

locals {
  # Define parameters for each workspace
  env = {
    default = {

      parameter_name = "/control_plane/${include.root.locals.environment.locals.workspace}/credentials"
      sharing_principals = ["ou-w3ow-k24p2opx"]
      tags = {
        Environment = "control-plane"
        Layer       = "Operations"
      }
    }
    "dev" = {

      create = true
    }
    "prod" = {

      create = true
    }
  }
  # Merge parameters
  environment_vars = contains(keys(local.env), include.root.locals.environment.locals.workspace) ? include.root.locals.environment.locals.workspace : "default"
  workspace = merge(local.env["default"], local.env[local.environment_vars])
}


terraform {
  source = "../../../modules/terraform-aws-ssm-parameter-sotre"

}

inputs = {
  parameter_name = "${local.workspace["parameter_name"]}"
  parameter_description = "Control plane credentials"
  parameter_type = "SecureString"
  parameter_tier = "Advanced"
  create_kms = true
  enable_sharing = true
  sharing_principals= local.workspace["sharing_principals"]
  parameter_value = jsonencode({
    cluster_name                       = dependency.eks.outputs.cluster_name,
    cluster_endpoint                   = dependency.eks.outputs.cluster_endpoint,
    cluster_certificate_authority_data = dependency.eks.outputs.cluster_certificate_authority_data,
    cluster_version                    = dependency.eks.outputs.cluster_version,
    cluster_platform_version           = dependency.eks.outputs.cluster_platform_version,
    oidc_provider_arn                  = dependency.eks.outputs.oidc_provider_arn,
    hub_account_id = split(":", dependency.eks.outputs.cluster_arn)[4]
  }  )
  tags = local.workspace["tags"]

}

Now, another stack is necessary, the IAM role to enable service account for argocd use the IAM authentication with spoke clusters. The module terraform-aws-iam/iam-eks-role allows to create the IRSA role but also is necessary create a custom policy to allow assume role in the spoke accounts. The Figure 2 depicts in depth this setup.

You can create simple stack for managing the role, or a module that supports the EKS definition and IAM role.

Figure 2. GitOps authentication summary.

So, the module is in modules/terraform-aws-irsa-eks-hub

modules/
├── terraform-aws-irsa-eks-hub
│   ├── README.md
│   ├── main.tf
│   ├── outputs.tf
│   └── variables.tf
└── terraform-aws-ssm-parameter-sotre
    ├── README.md
    ├── data.tf
    ├── main.tf
    ├── outputs.tf
    └── variables.tf

3 directories, 9 files

and the stack is:

#eks_role-terragrunt.hcl

include "root" {
  path = find_in_parent_folders("root.hcl")
  expose = true
}


dependency "eks" {
  config_path = "${get_parent_terragrunt_dir("root")}/infrastructure/containers/eks_control_plane"
  mock_outputs = {
    cluster_name = "dummy-cluster-name"
    cluster_endpoint = "dummy_cluster_endpoint"
    cluster_certificate_authority_data = "dummy_cluster_certificate_authority_data"
    cluster_version = "1.31"
    cluster_platform_version = "1.31"
    oidc_provider_arn =  "dummy_arn"
  }
  mock_outputs_merge_strategy_with_state = "shallow"
}
locals {
  # Define parameters for each workspace
  env = {
    default = {

      environment  = "control-plane"
      role_name    = "eks-role-hub"
      tags = {
        Environment = "control-plane"
        Layer       = "Networking"
      }
    }
    "dev" = {

      create = true
    }
    "prod" = {

      create = true
    }
  }
  # Merge parameters
  environment_vars = contains(keys(local.env), include.root.locals.environment.locals.workspace) ? include.root.locals.environment.locals.workspace : "default"
  workspace = merge(local.env["default"], local.env[local.environment_vars])
}


terraform {
  source = "../../../modules/terraform-aws-irsa-eks-hub"

}

inputs = {
  role_name = "${local.workspace["role_name"]}-${local.workspace["environment"]}"

  cluster_service_accounts = {
    "${dependency.eks.outputs.cluster_name}" = [
      "argocd:argocd-application-controller",
      "argocd:argo-cd-argocd-repo-server",
      "argocd:argocd-server",
    ]
  }
  tags = local.workspace["tags"]

}

Finally, the gitops_bridge stack must look like:

#eks_control_plane-terragrunt.hcl
include "root" {
  path = find_in_parent_folders("root.hcl")
  expose = true
}

include "k8s_helm_provider" {
  path = find_in_parent_folders("/common/additional_providers/provider_k8s_helm.hcl")
}

dependency "eks" {
  config_path = "${get_parent_terragrunt_dir("root")}/infrastructure/containers/eks_control_plane"
  mock_outputs = {
    cluster_name = "dummy-cluster-name"
    cluster_endpoint = "dummy_cluster_endpoint"
    cluster_certificate_authority_data = "dummy_cluster_certificate_authority_data"
    cluster_version = "1.31"
    cluster_platform_version = "1.31"
    oidc_provider_arn =  "dummy_arn"
  }
  mock_outputs_merge_strategy_with_state = "shallow"
}
dependency "eks_role" {
  config_path = "${get_parent_terragrunt_dir("root")}/infrastructure/iam/eks_role"
  mock_outputs = {
    iam_role_arn = "arn::..."
  }
  mock_outputs_merge_strategy_with_state = "shallow"

}
locals {
  # Define parameters for each workspace
  env = {
    default = {

      environment  = "control-plane"
      oss_addons = {
        enable_argo_workflows = true
        #enable_foo            = true
        # you can add any addon here, make sure to update the gitops repo with the corresponding application set
      }

      addons_metadata = merge(
        {
          addons_repo_url      = "https://github.com/gitops-bridge-dev/gitops-bridge-argocd-control-plane-template"
          addons_repo_basepath = ""
          addons_repo_path     ="bootstrap/control-plane/addons"
          addons_repo_revision = "HEAD"
        }
      )
      argocd_apps = {
        addons = file("./bootstrap/addons.yaml")
        #workloads = file("./bootstrap/workloads.yaml")
      }

      tags = {
        Environment = "control-plane"
        Layer       = "Networking"
      }
    }
    "dev" = {

      create = true
    }
    "prod" = {

      create = true
    }
  }
  # Merge parameters
  environment_vars = contains(keys(local.env), include.root.locals.environment.locals.workspace) ? include.root.locals.environment.locals.workspace : "default"
  workspace = merge(local.env["default"], local.env[local.environment_vars])
}


terraform {
  source = "tfr:///gitops-bridge-dev/gitops-bridge/helm?version=0.1.0"

}

inputs = {
  cluster_name                       = dependency.eks.outputs.cluster_name
  cluster_endpoint                   = dependency.eks.outputs.cluster_endpoint
  cluster_platform_version           = dependency.eks.outputs.cluster_platform_version
  oidc_provider_arn                  = dependency.eks.outputs.oidc_provider_arn
  cluster_certificate_authority_data = dependency.eks.outputs.cluster_certificate_authority_data

  cluster = {
    cluster_name =   dependency.eks.outputs.cluster_name
    environment  = local.workspace["environment"]
    metadata     = local.workspace["addons_metadata"]
    addons = merge(local.workspace["oss_addons"], { kubernetes_version = dependency.eks.outputs.cluster_version })

  }
  apps = local.workspace["argocd_apps"]
  argocd = {
    namespace = "argocd"
    #set = [
    #  {
    #    name  = "server.service.type"
    #    value = "LoadBalancer"
    #  }
    #]
    values = [
      yamlencode(
        {
          configs = {
            params = {
              "server.insecure" = true
            }
          }
          server = {
            "serviceAccount" = {
              annotations = {
                "eks.amazonaws.com/role-arn" = dependency.eks_role.outputs.iam_role_arn
              }

            }
            service = {
              type = "NodePort"
            }

            ingress = {
              enabled    = false
              controller = "aws"
              ingressClassName : "alb"


              aws = {
                serviceType : "NodePort"
              }

              annotations = {
                #"alb.ingress.kubernetes.io/backend-protocol" = "HTTPS"
                #"alb.ingress.kubernetes.io/ssl-redirect"                       = "443"
                #"service.beta.kubernetes.io/aws-load-balancer-type"            = "external"
                #"service.beta.kubernetes.io/aws-load-balancer-nlb-target-type" = "ip"
                #"alb.ingress.kubernetes.io/listen-ports" : "[{\"HTTPS\":443}]"
              }
            }
          }
          controller = {
            "serviceAccount" = {
              annotations = {
                "eks.amazonaws.com/role-arn" =  dependency.eks_role.outputs.iam_role_arn
              }

            }
          }
          repoServer = {
            "serviceAccount" = {
              annotations = {
                "eks.amazonaws.com/role-arn" =  dependency.eks_role.outputs.iam_role_arn
              }

            }
          }
        }
      )
    ]

  }

  tags = local.workspace["tags"]

}

Basically the main changes was the introduction to argocd map to setup the values for helm chart deployment to enable to use the IRSA role.

When you are using cross account deployment the profile that creates the secrets in hub cluster must to have access and permissions, for example in the repository the eks_control_plane stack introduce a new access entry:

#eks_control_plane-terragrunt.hcl
include "root" {
  path   = find_in_parent_folders("root.hcl")
  expose = true
}
dependency "vpc" {
  config_path = "${get_parent_terragrunt_dir("root")}/infrastructure/network/vpc"
  mock_outputs = {
    vpc_id = "vpc-04e3e1e302f8c8f06"
    public_subnets = [
      "subnet-0e4c5aedfc2101502",
      "subnet-0d5061f70b69eda14",
    ]
    private_subnets = [
      "subnet-0e4c5aedfc2101502",
      "subnet-0d5061f70b69eda14",
      "subnet-0d5061f70b69eda15",
    ]
  }
  mock_outputs_merge_strategy_with_state = "shallow"
}

locals {
  # Define parameters for each workspace
  env = {
    default = {
      create          = false
      cluster_name    = "${include.root.locals.common_vars.locals.project}-${include.root.locals.environment.locals.workspace}-hub"
      cluster_version = "1.32"

      # Optional
      cluster_endpoint_public_access = true

      # Optional: Adds the current caller identity as an administrator via cluster access entry
      enable_cluster_creator_admin_permissions = true
      access_entries = {
        #####################################################################################################################
        # Admin installation and setup for spoke accounts - Demo purpose- must be the ci Agent Role
        ####################################################################################################################
        admins_sso = {
          kubernetes_groups = []
          principal_arn     = "arn:aws:sts::123456781234:role/aws-reserved/sso.amazonaws.com/us-east-2/AWSReservedSSO_AWSAdministratorAccess_877fe9e4127a368d"
          user_name         = "arn:aws:sts::123456781234:assumed-role/AWSReservedSSO_AWSAdministratorAccess_877fe9e4127a368d/{{SessionName}}"

          policy_associations = {
            single = {
              policy_arn = "arn:aws:eks::aws:cluster-access-policy/AmazonEKSClusterAdminPolicy"
              access_scope = {
                type = "cluster"
              }
            }
          }
        }
      }

      cluster_compute_config = {
        enabled    = true
        node_pools = ["general-purpose"]
      }

      tags = {
        Environment = "control-plane"
        Layer       = "Networking"
      }
    }
    "dev" = {

      create = true
    }
    "prod" = {

      create = true
    }
  }
  # Merge parameters
  environment_vars = contains(keys(local.env), include.root.locals.environment.locals.workspace) ? include.root.locals.environment.locals.workspace : "default"
  workspace        = merge(local.env["default"], local.env[local.environment_vars])
}


terraform {
  source = "tfr:///terraform-aws-modules/eks/aws?version=20.33.1"

}

inputs = {
  create          = local.workspace["create"]
  cluster_name    = local.workspace["cluster_name"]
  cluster_version = local.workspace["cluster_version"]

  # Optional
  cluster_endpoint_public_access = local.workspace["cluster_endpoint_public_access"]

  # Optional: Adds the current caller identity as an administrator via cluster access entry
  enable_cluster_creator_admin_permissions = local.workspace["enable_cluster_creator_admin_permissions"]

  cluster_compute_config = local.workspace["cluster_compute_config"]

  vpc_id         = dependency.vpc.outputs.vpc_id
  subnet_ids     = dependency.vpc.outputs.private_subnets
  access_entries = local.workspace["access_entries"]


  tags = {
    Environment = include.root.locals.environment.locals.workspace
    Terraform   = "true"
  }

  tags = local.workspace["tags"]

}

The final Hub infrastructure is:

Figure 3. Control Plane Infrastructure.

👇 Get the code here.

velez94 / terragrunt_aws_gitops_blueprint

Public demo for Gitops bridge using terragrunt, OpenTofu and EKS

AWS GitOps Blueprint with Terragrunt

This project provides a blueprint for implementing GitOps on AWS using Terragrunt and Argo CD. It offers a structured approach to managing infrastructure as code and deploying applications across multiple environments.

The blueprint is designed to streamline the process of setting up a GitOps workflow on AWS, leveraging Terragrunt for managing Terraform configurations and Argo CD for continuous deployment. It includes configurations for essential AWS services such as EKS (Elastic Kubernetes Service) and VPC (Virtual Private Cloud), as well as GitOps components for managing cluster addons and platform-level resources.

Key features of this blueprint include:

Modular infrastructure setup using Terragrunt
EKS cluster configuration for container orchestration
VPC network setup for secure and isolated environments
GitOps bridge for seamless integration between infrastructure and application deployments
Argo CD ApplicationSets for managing cluster addons and platform resources
Environment-specific configurations for multi-environment deployments

Repository Structure

terragrunt_aws_gitops_blueprint/
├── common/
│

…

View on GitHub

Creating spoke Cluster Infrastructure

The second step is creating the spoke cluster infrastructure. 🧑🏾‍💻

To manage a spoke cluster a single terragrunt or tofu project is created independently for each team from a template, so, infrastructure has an individual pipeline to manage it, and each team could have custom CI/CD agents, also more flexibility to add features and components to the infrastructure stacks. In some cases, you can have a single pipeline to manage the infrastructure setup and work with environment or parameters managed by the orchestration CI/CD tool. This approach is utilized when necessary to have central governance and control, but consider the changes rates, common tasks and environment setup and CI/CD workers capacity assigned to central ops.

The code is like the hub repository; however, the main difference is in the stack GitOps bridge.

Let’s watch it in depth. 🕵️‍♀️

First, a new provider configuration is necessary, the cluster hub provider, in the terragrunt_aws_gitops_spoke_blueprint/common/additional_providers:

locals {
  workspace = get_env("TF_VAR_env", "dev")
  pipeline = "false"
  hub_account_id    = "105171185823"
}
generate "k8s_helm_provider" {
  path      = "k8s_helm_provider.tf"
  if_exists = "overwrite"
  contents = <<EOF
################################################################################
# Kubernetes Access for Spoke Cluster
################################################################################

# First, define the parameter store data source
data "aws_ssm_parameter" "hub_cluster_config" {
  count = 1
  with_decryption = true
  name  = "arn:aws:ssm:us-east-2:${local.hub_account_id}:parameter/control_plane/${local.workspace}/credentials"
 #"/control_plane/${local.workspace}/credentials"  # Adjust the parameter path as needed
}

provider "kubernetes" {
  host = try(jsondecode(data.aws_ssm_parameter.hub_cluster_config[0].value).cluster_endpoint, var.cluster_endpoint)
  cluster_ca_certificate = try(base64decode(jsondecode(data.aws_ssm_parameter.hub_cluster_config[0].value).cluster_certificate_authority_data), var.cluster_certificate_authority_data)

  exec {
    api_version = "client.authentication.k8s.io/v1beta1"
    command     = "aws"
    args = [
      "eks",
      "get-token",
      "--cluster-name",
      try(jsondecode(data.aws_ssm_parameter.hub_cluster_config[0].value).cluster_name, var.cluster_name),
      "--region",
      try(jsondecode(data.aws_ssm_parameter.hub_cluster_config[0].value).cluster_region, data.aws_region.current.name),
      "--profile",
      var.profile["${local.workspace}"]["profile"]
    ]
  }
  alias = "hub"
}

EOF
}

The gitops_bridge stack is:

#eks_control_plane-terragrunt.hcl
include "root" {
  path = find_in_parent_folders("root.hcl")
  expose = true
}

include "k8s_helm_provider" {
  path = find_in_parent_folders("/common/additional_providers/provider_k8s_hub.hcl")
}

dependency "eks" {
  config_path = "${get_parent_terragrunt_dir("root")}/infrastructure/containers/eks_spoke"
  mock_outputs = {
    cluster_name = "dummy-cluster-name"
    cluster_endpoint = "dummy_cluster_endpoint"
    cluster_certificate_authority_data = "dummy_cluster_certificate_authority_data"
    cluster_version = "1.31"
    cluster_platform_version = "1.31"
    oidc_provider_arn =  "dummy_arn"
  }
  mock_outputs_merge_strategy_with_state = "shallow"
}
locals {
  # Define parameters for each workspace
  env = {
    default = {

      environment  = "control-plane"
      oss_addons = {
        enable_argo_workflows = true
        #enable_foo            = true
        # you can add any addon here, make sure to update the gitops repo with the corresponding application set
      }

      addons_metadata = merge(
        {
          addons_repo_url      = "https://github.com/gitops-bridge-dev/gitops-bridge-argocd-control-plane-template"
          addons_repo_basepath = ""
          addons_repo_path     ="bootstrap/control-plane/addons"
          addons_repo_revision = "HEAD"
        }
      )


      tags = {
        Environment = "control-plane"
        Layer       = "Networking"
      }
    }
    "dev" = {

      create = true
    }
    "prod" = {

      create = true
    }
  }
  # Merge parameters
  environment_vars = contains(keys(local.env), include.root.locals.environment.locals.workspace) ? include.root.locals.environment.locals.workspace : "default"
  workspace = merge(local.env["default"], local.env[local.environment_vars])
}


terraform {
  source = "../../../modules/terraform-aws-gitops-bridge-spoke"

}

inputs = {
  cluster_name                       = dependency.eks.outputs.cluster_name
  cluster_endpoint                   = dependency.eks.outputs.cluster_endpoint
  cluster_platform_version           = dependency.eks.outputs.cluster_platform_version
  oidc_provider_arn                  = dependency.eks.outputs.oidc_provider_arn
  cluster_certificate_authority_data = dependency.eks.outputs.cluster_certificate_authority_data
  create_kubernetes_resources = false
  cluster = {
    cluster_name =   dependency.eks.outputs.cluster_name
    environment  = local.workspace["environment"]
    metadata     = local.workspace["addons_metadata"]
    addons = merge(local.workspace["oss_addons"], { kubernetes_version = dependency.eks.outputs.cluster_version })

  }
  hub_account_id =  include.root.locals.common_vars.locals.hub_account_id

  tags = local.workspace["tags"]

}

This stacks defines the data for cluster secret in hub cluster and metadata information for addons, the local module terraform-aws-gitops-bridge-spoke creates the access entry and for enable hub access using spoke role according to Figure 2 and reuse gitops-bridge-dev/gitops-bridge/helm with parameters to deploy the secret but not the argocd installation in spoke clusters. So the infrastructure composition for spokes IaC is:

Figure 4. Infrastructure composition.

An alternative approach is to manage the external secrets Operator and secrets manager, let a comment if you want to know how do it.

Finally, after runs the spoke stacks in Argocd Hub Server you can watch the clusters and metadata information:

Figure 5. Cluster and metadata Information.

For example some addons was deployed in spoke cluster using applications Set.

Figure 6. Applications Set in spoke clusters.

In the next post, you can learn how to custom addons and add advance setups. 🦸🦸

👇 Get the code here.

velez94 / terragrunt_aws_gitops_spoke_blueprint

Infrastructure for Spoke clusters blueprint for GitOps Bridge

AWS GitOps Scale Infrastructure with EKS Spoke Clusters

A comprehensive Infrastructure as Code (IaC) solution that enables scalable GitOps deployments on AWS using EKS clusters in a hub-spoke architecture with automated infrastructure provisioning and configuration management.

This project provides a complete infrastructure setup using Terragrunt and Terraform to create and manage AWS resources including VPC networking, EKS clusters, and GitOps tooling. It implements a hub-spoke architecture where a central hub cluster manages multiple spoke clusters through GitOps practices, enabling consistent and automated application deployments at scale.

The solution includes automated VPC creation with proper network segmentation, EKS cluster provisioning with secure configurations, and integration with GitOps tools through a bridge component that enables declarative infrastructure and application management.

Repository Structure


├── common/                          # Common configuration and variable definitions
│   ├── additional_providers/        # Provider configurations for Kubernetes, Helm, etc.
│   ├── common.hcl                  # Common Terragrunt configuration
│   ├── common.tfvars               #

…

View on GitHub

Thanks for reading and sharing! 🫶

GitOps and IaC at Scale – AWS, ArgoCD, Terragrunt, and OpenTofu – Part 1

Alejandro Velez — Thu, 13 Feb 2025 17:18:53 +0000

level 400

Welcome, builders! In this blog series, we will explore the exciting world of GitOps and Infrastructure as Code (IaC) at scale, focusing on the powerful combination of ArgoCD and Open Tofu and how those tools and frameworks can live together for improving your deployments and reduce the TOIL. The main purpose is to learn some patterns and considerations for increasing efficiency, improving the automation infrastructure provisioning without loss compliance and security requirements

Scenario

Suppose that you are a Cloud Engineer or DevOps engineer, and you have a mission, deploy a cloud native infrastructure to support a modern e-commerce application that offers the possibility to any buy and sell gems. You must implement a system to allow the operations at scale, keep the governance, use IaC and consider the least manual tasks for the setup and onboarding for the developers. The cloud provider for the organization is AWS and the main infrastructure services are EKS, RDS, VPC. Don’t forget that availability, security, and cost efficiency requirements.

In the previous blogs about IaC at scale the main questions and base guidelines was answered to apply the best practices, however, when talking about Kubernetes and cloud native apps many questions emerge from the shadows and suppose al challenge for the technical skills, for example:

What are the best practices to manage the cluster configuration at scale?
How can you connect both layers IaC and GitOps?
What are the GitOps topologies?
What are the best tools?
How can you improve developer experience?

Of course, with an AI assistant you can find pretty answers for those questions and some guides. But, we are builders and here we put a real-life example based on experiences and some challenges in a productive environment.

First, some theory and good practices 😎

The deployment patterns and GitOps Topology

If you want to know more about gitops, patterns, courses and more please visit opengitops

Key points based on the requirements:
• You must have multiple environments.
• Each environment must isolate from others.
• You must manage the cluster configuration at scale.

There are tree main Gitops topologies:
• Standalone topology
• Hub-Spoke Push Model
• Hub-Spoke Push and Pull model

For this scenario, the hub-spoke push model is the best solution. This setup involves deploying GitOps tools on a central "Hub" cluster. The Hub cluster acts as the control plane for GitOps, managing deployments to various clusters dedicated to different environments or workloads.

So, how can you deploy this topology in AWS using EKS?

First, let’s get some assumptions:
• Your accounts are in the same AWS organization.
• There is already a Networking layer configured.
• ArgoCD is the gitops tool.
• Github the VSC.
• There is no CI tool predefined, in this blog series Codepipeline and previous blog posts will be use.

Security considerations:

• The IAM roles and policies for your agents must be clear and apply the least privileged principle. Sometimes the infrastructure deployment role is different from orchestration deployment role.
• Use secrets manager to share the credentials between accounts using solid resource policy or use parameters store secure string with RAM. Both must be encrypted using KMS.
• Don’t expose your cluster endpoints to the internet, if it is necessary then restrict the access by IP range.
• The security of the pipeline must be a priority, the deployment agents can assume many privileged roles, so keep in mind to avoid security breaches.

Solution overview

Open-Source projects and tools:
Open Tofu
GitOps bridge
ArgoCD

AWS Services:

EKS
KMS
RAM
Parameter Store
ECR
IAM

The following diagrams shows the solution:

1- The code is versioned in git system, usually you have tree main kind of repositories: Infrastructure as code repositories, Argocd applications Set definitions, and app or microservices repositories. Many teams keep separated the Kubernetes manifests from code applications. This is a good practice due the code application and infrastructure definitions don’t follow the same lifecycle but other factors as mono repositories or multiple repositories are key factors. Just keep in different folders 😊.

2- The CI/CD service deploys the infrastructure components: the CI/CD agents deploy the control plane cluster, create shared resources to spoke AWS accounts, and create the argocd secret for hub cluster with the metadata to allow operations and bootstrapping configurations.

The agent’s role must be part of an access entry in EKS to allow this operation.

3- The shared resources are created are common practices that use secure string in parameter store and share it with RAM is more cost effective and practical for this setup, however, you could use secrets manager with resource policies.

4- The CI/CD service deploys the spoke resources: the agent deploys the spoke cluster and other resources. Here the agent can access the Hub Kubernetes cluster to create a secret for spoke cluster with the metadata to allow operations and bootstrapping configuration.

5- ArgoCD detects new cluster metadata and prepare to deploy de addons: Once the metadata is loaded the applications are deployed in the spoke or hub cluster, to accomplish this the Argocd application controller must assume the role in the spoke account.

In ArgoCD, the limit for metadata annotations is 262,144 characters (or 256 KB)

6- The application are created: here the deployments al created into the spoken cluster and Argocd sync with the desired state.

7- The cluster are ready to workloads.

It sounds pretty, but how can integrate both layers without overhead or manual operations?

The answer is to use the GitOps bridge framework: Is a community project that aims to showcase best practices and patterns for bridging the process of creating a Kubernetes cluster to subsequently managing everything through GitOps. It focuses on using ArgoCD or FluxCD, both of which are CNN-graduated projects.

gitops-bridge-dev / gitops-bridge

GitOps Bridge

The GitOps Bridge is a community project that aims to showcase best practices and patterns for bridging the process of creating a Kubernetes cluster to subsequently managing everything through GitOps. It focuses on using ArgoCD or FluxCD, both of which are CNCF-graduated projects.

For an example template on bootstrapping ArgoCD, see the GitHub repository GitOps Control Plane.

There are many tools available for creating Kubernetes clusters. These include "roll-your-own" solutions like kubeadm, minikube, and kind, as well as cloud-managed services like Amazon EKS. The method of cluster creation should not impact GitOps compatibility; GitOps engines should work with any tool that the user chooses for cluster creation. This includes scenarios where Kubernetes is used to create other Kubernetes clusters, such as with CAPI/CAPA, Crossplane, ACK, or any tool running inside Kubernetes to deploy Kubernetes.

The GitOps Bridge becomes extremely important in the context…

View on GitHub

Hands On

Creating a control plane

So, let’s hand in to the code for building a control plane.

1- The terragrunt project is created with the following structure:

The project also can be created with OpenTofu but here is an example using terragrunt a wrapper for opentofu and a powerful tool to keep DRY and manage IaC at scale.

In the infrastructure folder you can find two major stacks: network and containers. In containers stacks the cluster for control plane is deployed using the public aws modules and gitops bridge is deployed into the account using the official terraform module.

For example, the content for eks_control_plane stack:

#eks_control_plane-terragrunt.hcl
include "root" {
  path = find_in_parent_folders("root.hcl")
  expose = true
}
dependency "vpc" {
  config_path  = "${get_parent_terragrunt_dir("root")}/infrastructure/network/vpc"
  mock_outputs = {
    vpc_id         = "vpc-04e3e1e302f8c8f06"
    public_subnets = [
      "subnet-0e4c5aedfc2101502",
      "subnet-0d5061f70b69eda14",
    ]
     private_subnets                             = [
     "subnet-0e4c5aedfc2101502",
      "subnet-0d5061f70b69eda14",
      "subnet-0d5061f70b69eda15",
     ]
  }
  mock_outputs_merge_strategy_with_state = "shallow"
}

locals {
  # Define parameters for each workspace
  env = {
    default = {
      create       = false
      cluster_name = "${include.root.locals.common_vars.locals.project}-${include.root.locals.environment.locals.workspace}-hub"
      cluster_version = "1.31"

      # Optional
      cluster_endpoint_public_access = true

      # Optional: Adds the current caller identity as an administrator via cluster access entry
      enable_cluster_creator_admin_permissions = true

      cluster_compute_config = {
        enabled = true
        node_pools = ["general-purpose"]
      }

      tags = {
        Environment = "control-plane"
        Layer       = "Networking"
      }
    }
    "dev" = {

      create = true
    }
    "prod" = {

      create = true
    }
  }
  # Merge parameters
  environment_vars = contains(keys(local.env), include.root.locals.environment.locals.workspace) ?  include.root.locals.environment.locals.workspace : "default"
  workspace = merge(local.env["default"], local.env[local.environment_vars])
}


terraform {
  source = "tfr:///terraform-aws-modules/eks/aws?version=20.33.1"

}

inputs = {
  create= local.workspace["create"]
  cluster_name = local.workspace["cluster_name"]
  cluster_version = local.workspace["cluster_version"]

  # Optional
  cluster_endpoint_public_access = local.workspace["cluster_endpoint_public_access"]

  # Optional: Adds the current caller identity as an administrator via cluster access entry
  enable_cluster_creator_admin_permissions = local.workspace["enable_cluster_creator_admin_permissions"]

  cluster_compute_config = local.workspace["cluster_compute_config"]

  vpc_id = dependency.vpc.outputs.vpc_id
  subnet_ids = dependency.vpc.outputs.private_subnets

  tags = {
    Environment = include.root.locals.environment.locals.workspace
    Terraform   = "true"
  }

  tags = local.workspace["tags"]

}

Watch that the main properties for this stack are the cluster version and in this case the auto mode for EKS is enabled.

In the next blogs you can learn more about this!

For other hand, for gitops_bridge stack:

include "root" {
  path = find_in_parent_folders("root.hcl")
  expose = true
}


include "k8s_helm_provider" {
  path = find_in_parent_folders("/common/additional_providers/provider_k8s_helm.hcl")
}

dependency "eks" {
  config_path = "${get_parent_terragrunt_dir("root")}/infrastructure/containers/eks_control_plane"
  mock_outputs = {
    cluster_name = "dummy-cluster-name"
    cluster_endpoint = "dummy_cluster_endpoint"
    cluster_certificate_authority_data = "dummy_cluster_certificate_authority_data"
    cluster_version = "1.31"
  }
  mock_outputs_merge_strategy_with_state = "shallow"
}

locals {
  # Define parameters for each workspace
  env = {
    default = {

      environment  = "control-plane"
      oss_addons = {
        enable_argo_workflows = true
        #enable_foo            = true
        # you can add any addon here, make sure to update the gitops repo with the corresponding application set
      }

      addons_metadata = merge(
        {
          addons_repo_url      = "https://github.com/gitops-bridge-dev/gitops-bridge-argocd-control-plane-template"
          addons_repo_basepath = ""
          addons_repo_path     ="bootstrap/control-plane/addons"
          addons_repo_revision = "HEAD"
        }
      )
      argocd_apps = {
        addons = file("./bootstrap/addons.yaml")
        #workloads = file("./bootstrap/workloads.yaml")
      }

      tags = {
        Environment = "control-plane"
        Layer       = "Networking"
      }
    }
    "dev" = {

      create = true
    }
    "prod" = {

      create = true
    }
  }
  # Merge parameters
  environment_vars = contains(keys(local.env), include.root.locals.environment.locals.workspace) ? include.root.locals.environment.locals.workspace : "default"
  workspace = merge(local.env["default"], local.env[local.environment_vars])
}


terraform {
  source = "tfr:///gitops-bridge-dev/gitops-bridge/helm?version=0.1.0"

}

inputs = {
  cluster_name                       = dependency.eks.outputs.cluster_name
  cluster_endpoint                   = dependency.eks.outputs.cluster_endpoint
  cluster_platform_version           = dependency.eks.outputs.cluster_platform_version
  oidc_provider_arn                  = dependency.eks.outputs.oidc_provider_arn
  cluster_certificate_authority_data = dependency.eks.outputs.cluster_certificate_authority_data

  cluster = {
    cluster_name =   dependency.eks.outputs.cluster_name
    environment  = local.workspace["environment"]
    metadata     = local.workspace["addons_metadata"]
    addons = merge(local.workspace["oss_addons"], { kubernetes_version = dependency.eks.outputs.cluster_version })

  }
  apps = local.workspace["argocd_apps"]

  tags = local.workspace["tags"]

}

Here the addonsare enabled or disable using flags, in oss_addonsmap also you can and the ArgoCD application Set to bootstrap the cluster.

2- Create the application set for addons:

kind: ApplicationSet
metadata:
  name: cluster-addons
  namespace: argocd
spec:
  syncPolicy:
    preserveResourcesOnDeletion: true
  generators:
  - clusters:
      selector:
        matchLabels:
          environment: control-plane
  template:
    metadata:
      name: 'cluster-addons'
    spec:
      project: default
      source:
        repoURL: '{{metadata.annotations.addons_repo_url}}'
        path: '{{metadata.annotations.addons_repo_basepath}}{{metadata.annotations.addons_repo_path}}'
        targetRevision: '{{metadata.annotations.addons_repo_revision}}'
        directory:
          recurse: true
      destination:
        namespace: 'argocd'
        name: '{{name}}'
      syncPolicy:
        automated:
          allowEmpty: true

The key point here is the generators, where the cluster selector is just deploy the application in the clusters that have the label environment equal to control plane.
The source for de applications is the addons_repo_url defined in gitops bridge stack terragrunt_aws_gitops_blueprint/infrastructure/containers/gitops_bridge/terragrunt.hcl.

3- Connect to the cluster using a k8s client.
Setup the kube context using aws cli:

aws eks update-kubeconfig --name my-cluster --region us-east-1 --alias my-cluster-alias

For example:

aws eks update-kubeconfig --name gitops-scale-dev-hub --region us-east-2 --alias labvel-devsecops-hub --profile labvel-dev

Run:

k9s --context labvel-devsecops-hub

for interactive mode or just find the default ArgoCD server admin secret and run kubectl to create a forwarding to connect with the argocd server.

kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" --context labvel-devsecops-hub | base64 -d; echo

kubectl port-forward svc/argo-cd-argocd-server -n argocd 8080:443 --context labvel-devsecops-hub

Finally through the web browser you can find the argo apps:

The application Set:

And the cluster metadata:

In the next part you can learn more about enable and disable addons and apps and go deeper using AWS controllers.

You can find the public code here:

velez94 / terragrunt_aws_gitops_blueprint

Public demo for Gitops bridge using terragrunt, OpenTofu and EKS

AWS GitOps Blueprint with Terragrunt

Key features of this blueprint include:

Modular infrastructure setup using Terragrunt
EKS cluster configuration for container orchestration
VPC network setup for secure and isolated environments
GitOps bridge for seamless integration between infrastructure and application deployments
Argo CD ApplicationSets for managing cluster addons and platform resources
Environment-specific configurations for multi-environment deployments

Repository Structure

terragrunt_aws_gitops_blueprint/
├── common/
│

…

View on GitHub

Thanks for reading and sharing! ☺️⭐

DevSecOps with AWS- IaC at scale - Building your own platform – Part 3 - Pipeline as a Service

Alejandro Velez — Sat, 30 Nov 2024 23:30:51 +0000

Level 300

In the previous post for this series,DevSecOps with AWS- IaC at scale - Building your own platform - Part 1 and DevSecOps with AWS- IaC at scale - Building your own platform - Part 2 - CI for IaC you could learn more about creating the CI pipeline, key questions and practices to enable the self-service capabilities to reuse this pipeline. In the following example, you will learn a use case to allow self-services capabilities using the pipeline of pipeline approach and combine the service Catalog capabilities to get a Pipeline as a Service.

So, let’s create as Platform engineer. But first a little overview about the solution:

Solution Overview

Now review the solution exposed in the previous blog post. And focus on creating a SaaS model to deploy your standard pipelines in other DevSecOps accounts. Using CDK Pipelines for this approach. In the future post you can enable the self-service capabilities for now just focusing on the platform’s bases.

You need to follow the next steps:
1- Construct pipeline of pipelines using CDK Pipelines.
2- Test the pipeline of pipelines to create a pipeline for deployment in sandbox account.
3- Approve changes to pass to production environments.
4- Enable self service capabilities to add new project to the pipelines of pipelines.
5- Out the box, create self service portal according to IDP (Internal Developer Portal).

Requirements

• cdk >= 2.167.1
• AWS CLI >= 2.7.0
• Python >= 3.10.4

AWS Services

AWS Cloud Development Kit (CDK): is an open-source software development framework to define your cloud application resources using familiar programming languages.
AWS Identity and Access Management (IAM): Securely manage identities and access to AWS services and resources.
AWS IAM Identity Center (Successor to AWS Single Sign-On): helps you securely create or connect your workforce identities and manage their access centrally across AWS accounts and applications.
AWS CodeBuild: fully managed continuous integration service that compiles source code, runs tests, and produces software packages that are ready to deploy.
AWS CodePipeline: fully managed continuous delivery service that helps you automate your release pipelines for fast and reliable application and infrastructure updates.
AWS Key Management Service (AWS KMS): lets you create, manage, and control cryptographic keys across your applications and more than 100 AWS services.
AWS CloudFormation: Speed up cloud provisioning with infrastructure as code.
AWS Resource Access Manager: Simply and securely share your AWS resources across multiple accounts.

Hands On ☺️

First create a CDK pipelines Stack and deployment stage:

from aws_cdk import (
    Stage,
    Environment,
    Tags
)
from constructs import Construct

from ...cdkv2_dev_sec_ops_ia_c_terraform_stack import Cdkv2DevSecOpsIaCTerraformStack


class PipelineStageProd(Stage):

    def __init__(self, scope: Construct, id: str, props: dict = None,
                 compliance_buildef: dict = None,
                 terra_plan_buildef: dict = None,
                 kics_buildef: dict = None,
                 tfsec_buildef: dict = None,
                 checkov_buildef: dict = None,
                 testing_buildef: dict = None,
                 deployment_buildef: dict = None,
                 env_devsecops_account: Environment = None,
                 environments: dict = None,
                 tags:dict = None, **kwargs):
        super().__init__(scope, id, **kwargs)

        core_stack = Cdkv2DevSecOpsIaCTerraformStack(self, f"CoreDevSecOpsIaCTerraformStack-{props['project_name']}",
                                                     stack_name=f"CoreDevSecOpsIaCTerraformStack-{props['project_name']}",

                                                     props=props,
                                                     compliance_buildef=compliance_buildef,
                                                     tfsec_buildef=tfsec_buildef,
                                                     checkov_buildef=checkov_buildef,
                                                     testing_buildef=testing_buildef,
                                                     terra_plan_buildef=terra_plan_buildef,
                                                     kics_buildef=kics_buildef,
                                                     deployment_buildef=deployment_buildef,
                                                     env_devsecops_account=env_devsecops_account,
                                                     environments=environments

                                                     )

        for t in tags.keys():
            Tags.of(core_stack).add(key=t, value=tags[t])

Now, from main stack add the stages based on the project properties in yaml files, here a common question for manage parallel deployments and projects with this approach.

How can we deploy many stacks parallel when we have many projects?

The answer is using the CDK Pipelines waves capability, this capability is useful in several scenarios:

Parallel Deployments:
• Deploy multiple stages simultaneously to different regions or accounts.
• Reduce overall pipeline execution time by running independent deployments in parallel.
• Deploy the same application to multiple environments concurrently.
Deployment Organization:
• Group related deployments logically.
• Create deployment waves for different environments (dev, test, prod).
• Organize deployments by business units or applications.
Controlled Progressive Rollouts:
• Deploy to a subset of regions/accounts first.
• Implement progressive deployment patterns.
• Add validation steps between waves for safety.
Resource Optimization:
• Control concurrent deployments to manage resource usage.
• Balance deployment speed with system load.
• Optimize pipeline execution costs.
Dependency Management:
• Group stages that can run independently.
• Separate dependent deployments into sequential waves.
• Manage complex deployment orchestration.

While stages within a wave run in parallel, the waves themselves execute sequentially. This allows for controlled progression through your deployment pipeline while maximizing efficiency where parallel execution is beneficial.

Second, test the stack with the parameters, in this case all definitions in a folder project configs are loaded, and one pipeline is created for each parameters set (team or project definitions).

Each project properties are loaded with a python function and create a custom class to abstract the project configurations.

import os
from dataclasses import dataclass
from typing import Dict, Any, Optional
from aws_cdk import Environment
from .utils import load_yamls, load_yamls_to_dict, load_tags, find_yaml_files

@dataclass
class ProjectConfiguration:
    """Class to manage project configuration and environments"""

    def __init__(self, props_path: str, dirname: Optional[str] = None):
        """
        Initialize ProjectConfiguration

        Args:
            props_path (str): Path to the properties file
            dirname (str, optional): Base directory path. If None, uses the directory of this class
        """
        self.dirname = dirname if dirname is not None else os.path.dirname(__file__)
        self.props_path = props_path
        self.props = self._load_properties()
        self.environments = {}
        self.build_definitions = {}

        self._setup_environments()
        self._load_tags()
        self._load_build_definitions()

    def _load_properties(self) -> Dict[str, Any]:
        """Load main properties from YAML file"""
        full_path = os.path.join(self.props_path)
        return (load_yamls(full_path))[0]

    def _setup_environments(self) -> None:
        """Setup environment configurations"""
        self.devsecops_env = Environment(
            account=self.props['account_devsecops'],
            region=self.props['region_devsecops']
        )

        self.props["def_environments"] = {}
        for env_config in self.props["environments"]:
            self._process_environment(env_config)

        self.props["environments"] = self.environments

    def _process_environment(self, env_config: Dict[str, Any]) -> None:
        """Process individual environment configuration"""
        env_name = env_config["environment"]

        # Create CDK Environment
        self.environments[env_name] = Environment(
            account=env_config['deployment_account'],
            region=env_config['deployment_region']
        )

        # Basic environment configuration
        self.props["def_environments"][env_name] = {
            "deployment_region": env_config["deployment_region"],
            "deployment_account": env_config["deployment_account"],
            "enable": env_config["enable"],
            "manual_approval": env_config["manual_approval"],
            "peer_review_approval": env_config["peer_review_approval"]
        }

        # Test environment specific configuration
        if env_name == "test":
            self.props["def_environments"][env_name].update({
                "automate_testing": env_config["automate_testing"],
                "test_workspace": env_config["test_workspace"]
            })

        # Optional partner review email
        if "partner_review_email" in env_config:
            self.props["def_environments"][env_name]["partner_review_email"] = env_config["partner_review_email"]

    def _load_tags(self) -> None:
        """Load tags from YAML file"""
        tags = (load_yamls(os.path.join(self.props_path)))[1]['tags']
        self.props["tags"] = load_tags(tags)

    def _load_build_definitions(self) -> None:
        """Load all build definitions"""
        definition_mappings = {
            'compliance': 'path_def_compliance',
            'terra_plan': 'path_def_create_artifacts',
            'tfsec': 'path_def_tfsec',
            'checkov': 'path_def_checkov',
            'kics': 'path_def_kics',
            'testing': 'path_def_testing',
            'pipeline_cd': 'path_def_deploy'
        }

        for key, path_key in definition_mappings.items():
            self.build_definitions[key] = load_yamls_to_dict(
                os.path.join(self.dirname, self.props[path_key])
            )

    def get_environment(self, env_name: str) -> Environment:
        """Get specific environment configuration"""
        return self.environments.get(env_name)

    def get_build_definition(self, definition_type: str) -> Dict[str, Any]:
        """Get specific build definition"""
        return self.build_definitions.get(definition_type, {})


# Example usage with different configuration files
def get_project_configs():
    print("Loading project configs ...")
    dirname = os.path.dirname("./project_configs/environment_options/")
    dirname_pipes = os.path.dirname("./project_configs/pipeline_definitions")
    projects = find_yaml_files(dirname, "environment_options_terragrunt")

    project_configs= []
    for p in projects:
    # Create instances for different environments or configurations
        conf=ProjectConfiguration(dirname= dirname_pipes,props_path=p)
        project_configs.append(conf)
    return project_configs

Third, test in sandbox and add the appropriate approval steps.

Here code fragment for this deployment, here two waves are created according to reference diagram, the first deploy a sandbox template and after pass to production environments.

from aws_cdk import (
    Stack,
    Environment,
    Aws,
    aws_codecommit as codecommit,
    pipelines
)
from constructs import Construct
from .load_project_configs import get_project_configs
from .stages.deploy_pipeline_stack import PipelineStageProd


class Cdkv2DSOTerraformPipelineStack(Stack):

    def __init__(self, scope: Construct, construct_id: str, props: dict = None,

                 **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)


        repo = props['infra_repository']["name"]
        synth = pipelines.ShellStep(
            "Synth",



      input=pipelines.CodePipelineSource.connection(repo_string=f"{props['infra_repository']['owner']}/{repo}",
                                                          code_build_clone_output=True,

                                                          branch=props['infra_repository']['branch'],
                                                          connection_arn=f"arn:aws:codestar-connections:{Aws.REGION}:{Aws.ACCOUNT_ID}:connection/{props['infra_repository']['connection_id']}"
                                                          ),

            commands=[
                "npm install -g aws-cdk",  # Installs the cdk cli on Codebuild
                "pip install -r requirements.txt",
                "pip install checkov",
                # Instructs Codebuild to install required packages
                "npx cdk synth",
            ]
        )

        pipeline = pipelines.CodePipeline(self, f"CDKPipeline-{props['project_name']}",
                                          self_mutation=True,
                                          cross_account_keys=True,
                                          synth=synth,
                                          pipeline_name=repo,

                                          )

        sandbox_wave = pipeline.add_wave("DeploySandbox")
        prod_wave = pipeline.add_wave("DeployProd")


        # add manual approval for prod wave
        sandbox_wave.add_post(pipelines.ManualApprovalStep("PromoteToProd",
                                                           comment="Ready to apply to prod deployments"))
        projects = get_project_configs()
        for project in projects:
            project_name = project.props.get("project_name")
            print(project_name)

            stage_id = f"Deploy-{project_name}"
            # create wave
            deploy_stage = PipelineStageProd(self, stage_id, stage_name=stage_id,
                                             props=project.props,
                                             compliance_buildef=project.get_build_definition("compliance"),
                                             tfsec_buildef=project.get_build_definition("tfsec"),
                                             checkov_buildef=project.get_build_definition("checkov"),
                                             testing_buildef=project.get_build_definition("testing"),
                                             terra_plan_buildef=project.get_build_definition("terra_plan"),
                                             kics_buildef=project.get_build_definition("kics"),
                                             deployment_buildef=project.get_build_definition("pipeline_cd"),
                                             env=project.devsecops_env,  # env_client_prd_account,
                                             env_devsecops_account=project.devsecops_env,
                                             environments=project.environments,
                                             tags=project.props.get("tags")
                                             )  # env=env_client_prd_account)
            if project_name.endswith("sandbox"):
                sandbox_wave.add_stage(
                    deploy_stage,

                )
            else:
                prod_wave.add_stage(
                    deploy_stage
                )

You can modify it according to your requirements.

The AWS console view is something like this:

Finally, you are ready to deploy more projects, in this case the project structure allow put a file for each project in the path and the cdk app automatically extend and create new pipelines:


tree project_configs/environment_options/
project_configs/environment_options/
├── environment_options_advanceiac_v2.yaml
├── environment_options_ecs_fargate_pattern.yaml
├── environment_options_project_1.yaml
├── environment_options_project_2.yaml
├── environment_options_project_3_networking.yaml
├── environment_options_project_4_shared.yaml
├── environment_options_terragrunt_blueprint.yaml
├── environment_options_terragrunt_sandbox.yaml
└── environment_options_another_project.yaml

1 directory, 9 files

Thanks for reading and sharing. If you want to know more and get the code please follow me and at the end of this series I will share the final project version as opensource project.

DEV Community: Alejandro Velez

Modernizing Cross-Account Networking: From VPC Peering Complexity to 5 VPC Lattice Resources - NetDevOps with AWS

🎯 The Challenge

Cloud Networking at Scale is Broken (And We Keep Pretending It's Not)

What is NetDevOps?

My Approach to the Challenge

The Architecture

How I Organized the Code

Hands-On: Let's Build It

Step 1: The VPC (Minimal Attack Surface)

Step 2: Security Groups (This Is Where It Gets Interesting)

Step 3: Aurora PostgreSQL

Step 4: VPC Lattice (Here's Where the Magic Happens)

Cross-Account Sharing with RAM (and Resharing!)

Step 5: Network Testing as Code (My Favorite Part)

🔄 Terragrunt Orchestration

The Security Model

How the Consumer Connects

What I Learned

Cost and the Path to a "Networkless" World

🔮 What's Next: The Deep Traffic Inspection Challenge

velez94 / tofu-aws-netdevops-rds-external

Template project to show how to use vpc lattice and cross account access with RAM and Databases in AWS

NetDevOps External - AWS Infrastructure with VPC Lattice

Architecture

📚 Resources

Automating AWS Well-Architected Reviews with Kiro CLI

The Problem: Manual Assessments Don't Scale

The Ecosystem: Tools That Accelerate WAFR Today

The Solution: WAFR Assessment Scaffold

Architecture Overview

Prerequisites: AWS CLI Profile Configuration

Option 1: Specify in the prompt (recommended)

Option 2: Set in MCP server environment

Client profile documentation

Minimum IAM permissions required

Hands On

Setting Up the Scaffold

velez94 / aws_well_architected_assement_scaffold

Well Architected Assement Scaffold Project with AI and Human on the loop

Well-Architected Assessment Scaffold

What's Included

.kiro/ - Kiro AI Configuration

config/ - Project Configuration

docs/ - Documentation

The Agent Architecture

Native Orchestration with Kiro Subagents

How Subagents Get Triggered

The 4-Phase Assessment Flow

Pillar Configuration with Weights

Running a Pillar Assessment with Kiro CLI

Multi-Client Scalability

Integrating with the WA Tool API: Start Fresh or Continue an Existing Review

Leveraging Trusted Advisor Integration

The Gap Analysis: Claims vs. Reality

Compliance Integration

GenAI Patterns Applied

Human-in-the-Loop (HITL)

Retrieval-Augmented Generation (RAG)

Agentic Orchestration (Fan-out / Fan-in)

Tool-Augmented Generation

Supervised Autonomy

Skills: Deep Knowledge On-Demand

How It Works

Skills vs. Prompts — When to Use Each

The Scaffold's Skill Library

Knowledge Bases: Searching Through Client Data

How It Works

The Three-Layer Context Strategy

Best Practices and Lessons Learned

Here my new post using Kiro and MCP to improve and modernize my current workflow for Authorization pipeline using Identity Center and IAM Access Analyzer

Re-imagine DevSecOps with AWS - CD applied to Authorization with IAM Identity Center and AWS IAM Access Analyzer

Alejandro Velez for AWS Community Builders ・ Mar 7

Re-imagine DevSecOps with AWS - CD applied to Authorization with IAM Identity Center and AWS IAM Access Analyzer

The Challenge

Key Components

Workflow Summary

Security and Governance Highlights

Hands On

The results

`.kiro/` - Kiro AI Configuration

`config/` - Project Configuration

`docs/` - Documentation