DEV Community: Pranit Raje

How to Import Existing AWS Infrastructure into Terraform Without Downtime

Pranit Raje — Wed, 07 Oct 2026 07:11:11 +0000

Bringing a live three-tier app under Terraform control with Terraform Search, HCP Terraform's Search and Import, Terraform MCP Server, HashiCorp Agent Skills and Sentinel policy that stopped a plan from replacing my EKS cluster

Most organisations have at least one of these: an application that is genuinely important, genuinely in production, and genuinely not in code. Nobody planned it that way. Someone spun it up in the console, a team started depending on it, then a customer did, and now there is a quiet understanding that nobody touches it unless something is on fire.

This is the real IaC adoption problem. Greenfield is easy. The hard part is the brownfield stuff that already works and therefore cannot be rebuilt just so it can be described properly.

Terraform has had an answer for years — terraform import, and later the declarative import block. Both work. Both are a grind at any real scale. One import block per resource, hand-written config to match whatever is actually deployed, then plan, read the diff, fix a field, plan again. For a forty-resource app that is a tense sprint, because every plan against a live system is a small act of faith.

So when Terraform shipped Search (list blocks in .tfquery.hcl) and HCP Terraform shipped Search and Import, I wanted to know whether the economics had actually changed. Not on a toy S3 bucket — on something with the shape of a real app: load balancer in front, compute in the middle, database at the back, and all the networking and IAM plumbing holding it together.

A note on what this is

This was a demo I built for a few clients while working at HashiCorp, to show what HCP Terraform, Search and Import, the Terraform MCP Server, and HashiCorp's official Agent Skills can do together. It is not production-ready, and there is plenty of room to improve it.

Every environment is different. Account layouts, tagging discipline, naming conventions, and blast-radius tolerance all vary. Take the learnings here, not the artefacts, and adapt them — you will likely find better approaches for your own setup. Search and Import is also still an experimental feature as of this date, which is worth factoring into how much you lean on it.

The workflow

Everything below is one box in this picture.

Four moving parts:

Terraform Search does discovery. I declare what I am looking for instead of clicking through the console.
The Terraform MCP Server and HashiCorp's Terraform Agent Skills do the authoring. The assistant writes the query file and later fixes the generated config, but every argument comes from the official registry docs rather than from memory.
HCP Terraform's Search and Import turns discovered resources into a starter configuration plus matching import blocks, in bulk.
Sentinel policies sit on the workspace so no run can quietly turn an import into a rebuild.

That fourth one is the reason this was safe enough to attempt at all.

The application

Deliberately boring, because the interesting part is the import workflow.

Tier	AWS services
Front end	Application Load Balancer, target group, HTTP listener
Web / app	EKS cluster and managed node group, launch template, IAM roles, IRSA OIDC provider, ECR repository
Data	DynamoDB table
Foundation	VPC, 4 subnets across 2 AZs, internet gateway, NAT gateway, EIP, 2 route tables, routes, subnet associations, 3 security groups and their ingress rules

A small Flask app. User fills in a form, the app writes it to DynamoDB, a list view reads it back.

That screenshot matters more than it looks like it should. Through everything that follows, this page stayed up. That was the actual success criterion — not "did apply exit zero" but "did the thing I imported keep serving users while I imported it".

I provisioned the stack with a CloudFormation template, purely to create the brownfield condition. It could equally have been a console session, a Python script, or a runbook full of aws CLI commands. The only property I needed was that Terraform had never seen it and had no state for it.

The one convenience CFN gave me is that it tags everything with aws:cloudformation:stack-name, which made a clean discovery handle. If your brownfield app has no such tag, you will lean harder on names, VPC IDs, and parent-resource scoping — which I had to do for plenty of resources anyway.

Step 1: Discovery as code

Terraform Search works through list blocks. Each one says "find resources of this type matching this filter". You run terraform query, and Terraform returns what it found, including the identity attributes needed to import each resource.

Small feature, but it changes the character of the work. Discovery stops being archaeology you do by hand and becomes a file you can commit, review, and re-run.

Before you write anything, check that your resource types are actually supported. List resource coverage is per-provider and per-version, and it is growing fast, so anything written down about it goes stale quickly. The reference I use every time is this auto-generated report, which tracks supported list resources and refreshes as providers ship updates. Cheaper than discovering mid-build that the resource you care about has no list support yet. You can also query your own provider schema directly:

terraform providers schema -json | \
  jq '.provider_schemas | to_entries | map({provider: .key, lists: (.value.list_resource_schemas // {} | keys)})'

Writing the query file is fiddly, though. Twenty-three resource types, each with its own filter semantics, and the argument names are not guessable. aws_vpc takes an EC2-style filter { name = "tag:..." }. aws_launch_template takes launch_template_names. aws_eks_node_group takes cluster_name. aws_iam_role takes nothing and returns every non-service-linked role in the account.

Exactly the kind of work where an AI assistant helps, and exactly the kind where it will confidently invent an argument that does not exist. So I constrained it with two things: the terraform-search-import Agent Skill (Search conventions and workflow) and the Terraform MCP Server (live access to registry docs). Then I made the lookup non-optional:

# Process
1. **Inventory**: Parse `three-tier-app.yaml` and list every resource
   (logical ID + `Type`).
2. **Map each resource to a list resource**:
   a. Map each CloudFormation type to its Terraform resource type in `hashicorp/aws`.
   b. Use the Terraform MCP server to check whether the latest `hashicorp/aws`
      provider offers a **list resource** for that type. Check the registry docs;
      don't rely on memory.
   c. If, and only if, `aws` has no list resource for it, check `hashicorp/awscc`.
   d. If neither provider supports it, don't invent one. Record it as unsupported.
3. **Filter by tag**: If a list resource can't filter by tag, use the narrowest
   documented alternative (e.g., VPC ID, name prefix, parent resource).
   Add an HCL comment that explains why you used it.

# Constraints
- Every list resource and argument must exist in the registry docs you looked up.
  Never guess.

Three instructions did most of the work.

"Don't rely on memory." On an earlier attempt, before I was this explicit, one filter argument came back from what was effectively a general web search. It looked plausible. It was wrong.

"Don't invent one. Record it as unsupported." List resource coverage is expanding fast but incomplete. In the end there was one gap: AWS::EC2::VPCGatewayAttachment has no aws list resource, so it fell back to awscc.

"Add an HCL comment that explains why." The most valuable line in the prompt. The query file now documents its own compromises.

Clean, tag-filtered discovery:

list "aws_vpc" "stack_vpcs" {
  provider         = aws
  include_resource = true

  config {
    filter {
      name   = "tag:aws:cloudformation:stack-name"
      values = [var.stack_name]
    }
  }
}

And the compromises, which are the more honest half of the file:

# Route resources cannot be tagged; discover routes beneath the tagged tables.
list "aws_route" "routes_in_stack_route_tables" {
  for_each         = toset([for table in list.aws_route_table.stack_route_tables.data : table.state.id])
  provider         = aws
  include_resource = true

  config {
    route_table_id = each.value
  }
}

# The EKS cluster list resource has no filter arguments and lists clusters region-wide.
list "aws_eks_cluster" "eks_clusters" {
  provider         = aws
  include_resource = true
}

If you take one piece of syntax from this post, take the for_each in that first block. Child resources — routes, route table associations, security group rules, listeners — generally cannot be tagged, so you cannot find them by tag. But you can find their tagged parents and pivot: feed the parents' IDs into a for_each on the child's list block. list.aws_route_table.stack_route_tables.data is the output of one query used as input to the next. Discovery composes.

That pivot is also why include_resource = true appears on every block. By default a query returns only resource identities. Set it to true and the provider returns full resource objects, which is what makes table.state.id readable by the next query. Without it the for_each has nothing to pivot on. The docs note it may affect performance, so it is worth being deliberate about rather than setting everywhere out of habit — though for a few dozen resources I never noticed the cost.

The resources with no filters at all (EKS clusters, load balancers, DynamoDB tables, ECR repositories, IAM roles) return everything in the region or account. Fine in a demo account. Noisy in a busy shared one — a real limitation, and one more reason the phased approach below is worth the extra runs.

Twenty-three list blocks. One terraform query. Every resource found, with its import identity attached.

Step 2: From discovery to configuration

With the query file validated, I moved to HCP Terraform's Search and Import. Point it at the query, it runs discovery, you tick the resources you want, and it generates a starter configuration.

Have your resource IDs ready before you open this screen

This is the prerequisite nobody tells you about, and it caught me out on my first attempt.

Search and Import returns everything your query matched — resources already managed by Terraform alongside the unmanaged ones you are trying to adopt. It is a discovery tool, not a filter. Selecting the right rows is your job.

Say you want to import your unmanaged security groups. Your list block matches on the VPC or a tag, and you get back a list of every security group in scope. Some are already in state. Some belong to a different team. Some are the ones you actually want. To tell them apart on that screen you need the exact security group IDs of your targets in hand — sg-0bd9f4d46a38xxxxx, not "the one for the EKS nodes".

Two things follow from that:

Build your inventory first. Before you touch the UI, have a list of the resources you intend to import and their unique identifiers. Console, aws CLI, a tagging report, your CMDB — however you get there, get there before you start selecting. Trying to assemble that list while staring at a few hundred discovered rows is how you import something that belongs to somebody else.

Know which identifier each resource type uses, because it is not consistent. A security group is identified by id, an EKS cluster by name, a load balancer or IAM policy by arn, an EKS node group by cluster_name plus node_group_name. The identity schema is documented per resource in the Terraform Registry — check it there rather than assuming, which is another spot where having the MCP Server wired in paid off. You can see the shape of it in the generated import blocks:

# Security group — identified by id
identity = {
  account_id = "1243XXXXXXXX"
  id         = "sg-0bd9f4d46a38XXXXX"
  region     = "ap-southeast-1"
}

# EKS node group — composite identity, no id at all
identity = {
  account_id      = "1243XXXXXXXX"
  cluster_name    = "demo-eks-cluster"
  node_group_name = "demo-eks-nodegroup"
  region          = "ap-southeast-1"
}

This gets worse the busier your account is. In a demo account with one stack, the discovered list is short and obvious. In a shared production account where list "aws_iam_role" returns every role you own, the identifiers are the only thing standing between you and adopting a resource another team depends on.

The same thing works from the CLI:

terraform init
terraform query                                  # discover
terraform query -generate-config-out=main.tf     # generate resource + import blocks

For each resource you get a resource block populated from the live API response, plus an import block keyed on identity:

import {
  to = aws_vpc.stack_vpcs_0
  identity = {
    account_id = "1243XXXXXXXX"
    id         = "vpc-037c4dd02c90XXXXX"
    region     = "ap-southeast-1"
  }
}

resource "aws_vpc" "stack_vpcs_0" {
  cidr_block                           = "10.0.0.0/16"
  enable_dns_hostnames                 = true
  enable_dns_support                   = true
  enable_network_address_usage_metrics = false
  instance_tenancy                     = "default"
  region                               = "ap-southeast-1"
  tags = {
    Name = "demo-vpc"
  }
}

Previously I would have written that import block by hand, looked up the ID, hand-written the resource block, and iterated on plan until the diff was empty — forty times. Here, forty resources' worth of scaffolding arrived at once with correct identities.

That is the real win, and I want to state it plainly: Search and Import does not give you production-ready Terraform. It gives you a very good head start on the mechanical half of the job.

Step 3: "Starter configuration" means exactly that

HCP Terraform is upfront about it. The feature is experimental, and the console tells you to review and modify this configuration before applying. Having worked through it, that label is precise.

The reason is structural. The generator populates config from everything the API returned, but a provider schema mixes three kinds of fields: things you set, things the API computes, and things that are mutually exclusive with other things. The API response does not distinguish between them. So you get attributes that cannot legally be set, pairs of conflicting arguments, and a lot of zero-valued noise.

terraform validate on the raw file produced a long list of errors. The patterns repeat, so they are worth naming:

Read-only attributes written as inputs. tags_all is the classic — computed from tags plus provider default tags.
= null and zero-value noise. disk_size = 0, read_capacity = 0 on a PAY_PER_REQUEST DynamoDB table, empty timeouts blocks.
Mutually exclusive arguments both set. The biggest category. launch_template.id and launch_template.name. max_unavailable and max_unavailable_percentage. subnets and subnet_mapping on an LB.
Values outside valid ranges. order = 0 on a listener action, where the provider needs 1 or higher.
Inline vs standalone conflicts. Subtle and important: the generator emitted inline ingress/egress blocks on aws_security_group and standalone aws_vpc_security_group_ingress_rule resources for the same rules, because the query discovered both. Those cannot coexist. Same story with inline route blocks vs aws_route.
Redundant provider = aws lines, which validate rejects inside import blocks.

Rather than fix 1,300 lines by hand, I pointed the same MCP-grounded setup at the generated file:

# Objective
Find and fix broken or invalid code in `main.tf`. Keep the intended
infrastructure exactly as it is.

# Process
1. **Baseline**: Run `terraform validate`. Record every error and warning.
2. **Documentation review**: Use the Terraform MCP server to check that every
   argument is valid in the schema, that conflicting arguments aren't set
   together (`ConflictsWith` / `ExactlyOneOf`), and that read-only attributes
   are not being set.
3. **Classify** each finding as Breaking or Warning, based on a specific doc
   page or validate error. Never guess.
4. **Fix**: smallest change that works. If a fix could force a resource to be
   replaced, or the right value is unclear, stop and ask.
5. **Re-validate** and repeat until validate passes or the rest needs my input.

The two instructions I would not drop: "keep the intended infrastructure exactly as it is" and "if a fix could force a replacement, stop and ask." An assistant optimising for a clean validate will happily delete an argument to make an error go away. During a brownfield import, deleting an argument is how you end up with a plan that wants to replace your production node group. Correctness here is not "validate passes" — it is "validate passes and the plan is a pure import".

terraform validate only checks schema and syntax. It cannot tell you whether your plan is about to replace a live resource. A clean validate is the start of the review, not the end of it.

So I needed something that could tell me that, on every run, before anything was applied.

Step 4: Guardrails first, imports second

I want to be emphatic about the ordering, because it is the most transferable idea here.

I attached the policies before I imported anything.

The failure mode of a brownfield import is not subtle, but it is quiet. You misread one attribute. That attribute turns out to be ForceNew. Your plan, which you believed was an import, now contains a replace. Reading a forty-resource plan at the end of a long day, you can absolutely miss that — and the blast radius is your running application.

That is a textbook policy-as-code problem: a specific, machine-checkable invariant that must hold on every run.

Policy one: no destructive changes (hard-mandatory)

An import run must not destroy or replace anything. In plan terms, three action sets are forbidden: ["delete"], ["delete","create"], and ["create","delete"]. Everything else — imports, no-ops, updates — is fine as far as this policy is concerned.

import "tfplan/v2" as tfplan
import "tfrun"

# Skip enforcement for explicit destroy runs.
is_destroy_run = tfrun.is_destroy

destructive_actions = [
    ["delete"],
    ["delete", "create"],
    ["create", "delete"],
]

destructive_changes = filter tfplan.resource_changes as _, rc {
    not is_destroy_run and any destructive_actions as da {
        actions_equal(rc.change.actions, da)
    }
}

main = rule {
    is_destroy_run or violation_count is 0
}

Two details worth copying. The tfrun.is_destroy escape hatch means terraform destroy still works when you genuinely mean it — a guardrail that blocks legitimate operations gets switched off, and a switched-off guardrail protects nothing. And the policy prints the address, type, and action of every offending resource, so a failure tells you why without hunting.

This runs at hard-mandatory. No override. I cannot talk myself past it at 7pm.

Policy two: flag in-place updates (soft-mandatory)

A correct import produces a plan full of imports and nothing else. If the plan also wants to update something, that is not dangerous but it is always informative — it means my generated config does not exactly match what is deployed. Either the generator got a field slightly wrong, or there is real drift.

I do not want to block that. I want to be told and then decide.

updated_resources = filter tfplan.resource_changes as _, rc {
    any update_actions as ua {
        actions_equal(rc.change.actions, ua)
    }
}

main = rule {
    update_count is 0
}

policy "no-destructive-changes" {
  source            = "./policies/no-destructive-changes.sentinel"
  enforcement_level = "hard-mandatory"
}

policy "flag-resource-updates" {
  source            = "./policies/flag-resource-updates.sentinel"
  enforcement_level = "soft-mandatory"
}

Two policies, about 120 lines of Sentinel. That is the entire safety apparatus.

Step 5: Import in phases

The temptation, once you have a working query file and generated config, is to apply all forty resources in one run. I did not, and I would push back on anyone who wanted to. I imported in dependency order, validating each layer before starting the next:

Networking — VPC, subnets, gateways, route tables, routes, associations
IAM — roles, inline policies, customer-managed policies, OIDC provider
Security groups — and their ingress rules
Compute — launch template, EKS cluster, node group
Load balancing — target group, ALB, listener
Data — DynamoDB table
Registry — ECR repository

Networking:

IAM:

Compute:

Why this is worth the extra runs:

Blast radius is one layer. If something looks wrong in the security group phase, there are three resources to inspect, not forty across seven services.

Dependencies resolve naturally. The EKS cluster references subnet IDs, security group IDs, and a role ARN. By the time I reach compute, those are already in state and verified.

Plans stay readable. This is the one I care about most. A plan with five imports is a plan I will actually read. A plan with forty is a plan I will scroll past while telling myself it looks fine. The policies exist because I do not fully trust myself on that — but the best way to use a guardrail is to not need it, and small plans are how you get there.

The full sequence in the workspace history, each entry one layer, reviewed and applied on its own:

Where the guardrails earned their keep

They fired twice. Both times they were right.

The one that would have cost me my EKS cluster

The starter configuration for my cluster came back with this:

resource "aws_eks_cluster" "eks_clusters_0" {
  bootstrap_self_managed_addons = false
  # ...
}

One boolean, generated for me, with a value that looks like a sensible default. But bootstrap_self_managed_addons is ForceNew on aws_eks_cluster, because it describes a decision made at creation time. My live cluster was created with it true. A mismatch on that field does not produce an update — it produces a replacement.

So a plan I thought was importing my EKS cluster was proposing to destroy and recreate the cluster that was, at that moment, serving the app in the screenshot at the top of this post.

The hard-mandatory policy caught it and failed the run:

This is the whole argument, so it is worth sitting with. I did review the config. terraform validate passed cleanly — false is a perfectly valid value for that argument, the right type, in the right place. Nothing about the line was wrong in isolation; it just did not match reality. The only way to catch it by eye was to already know that this specific boolean is ForceNew, and then to spot the # forces replacement marker in several hundred lines of plan output for a resource with a dozen nested blocks.

A policy reading the plan's action list needs none of that context. It does not need to know what bootstrap_self_managed_addons means. It needs to know that ["delete","create"] is not allowed during an import. That is the strength of policy as code — it checks a structural invariant instead of relying on my attention span.

The fix was a one-character change: flip it to true to match the cluster as deployed, re-run, and the plan became a clean import. Found by a guardrail instead of by an incident.

The one that was just drift

During the security group phase, the soft-mandatory policy spoke up. A few in-place update actions — not destructive, just minor differences between my generated config and the live rules.

Exactly the behaviour I wanted. The run paused, told me which resources it intended to modify, and waited. I checked each one, confirmed they were benign, overrode deliberately, and applied.

The difference in handling between these two cases is the point of having enforcement levels. hard-mandatory for "this must never happen", soft-mandatory for "a human needs to look at this". Collapse both into a blocking rule and the import becomes unworkable. Collapse both into a warning and the EKS replacement goes through.

The result

Every component of the app is now in Terraform state and managed through HCP Terraform. Around 40 resources across 23 types, including the one that needed awscc. About 1,300 lines of validated HCL, with the import blocks as a precise record of what was adopted and when.

And the app never went down. Not because I was careful, though I tried to be, but because the system was set up so that carelessness would be caught.

What I had at the end is a managed application. What I did not have is well-structured code.

Next: a 1,300-line `main.tf` is not where this ends

Let me be honest about what came out of this, because "I imported it successfully" papers over a lot.

The file is a monolith. Every ID is hardcoded — vpc-037c4dd02c90XXXXX appears a dozen times, security group IDs are inlined into the EKS cluster's vpc_config, role ARNs are string literals. Resource names are generator artefacts (stack_subnets_0, iam_roles_32, ingress_rules_in_stack_groups_1_5) that describe the query that found them, not what they do. The region repeats on nearly every resource.

This is a reasonable place for an import to end. It is an unreasonable place to stay. You cannot reuse any of it for a second environment, and the hardcoded IDs mean Terraform does not actually know the subnets belong to the VPC — it just happens to have the right strings in the right places.

So the next step is refactoring, and I plan to use HashiCorp's Agent Skills again for it, since they encode HashiCorp's own module design best practices:

Decompose into network, iam, security, eks, alb, data, registry
Replace every hardcoded ID with a real reference, so the dependency graph reflects actual dependencies
Promote region, environment name, and sizing to variables
Name resources after their purpose, not their discovery order
Define proper inputs and outputs at each module boundary

The ID-to-reference conversion is the bulk of the work and the highest-value part, because it is what turns a snapshot of your infrastructure into a description of it. That is a post of its own.

The framing I have settled on: import gets you management; refactoring gets you Infrastructure as Code. Two different milestones, and worth being clear-eyed about which one you have reached.

What I would tell you before you try this

Discovery is now a reviewable artefact. Moving from "paste IDs into a spreadsheet" to "a committed .tfquery.hcl file" is a bigger shift than it looks. When someone adds a resource next month, I re-run the query instead of repeating the archaeology.

Check support before you plan the work, not during it. List resource coverage varies by provider and version. The supported-resources report takes a minute to check and tells you whether this workflow is even available for the resources you care about.

Bring your resource IDs to the party. Search and Import shows you managed and unmanaged resources together. Picking the right ones out of that list requires knowing their exact identifiers beforehand — and knowing which identifier each resource type uses, because it varies. Do that inventory before you open the UI.

Learn the parent-pivot pattern. Most child resources cannot be tagged. for_each over a parent list block's output is how you reach them. That one pattern unlocked routes, associations, SG rules, and listeners.

Ground your AI assistant in the registry, or don't use it. An Agent Skill for conventions plus the MCP Server for version-specific docs is what made AI assistance trustworthy rather than a source of plausible-looking errors. "Check the docs, never guess, record what you couldn't cover" belongs in your prompt verbatim.

Read "starter configuration" literally. The generator writes back everything the API returned — computed attributes, conflicting arguments, zero-value noise. Budget real time for cleanup. Still dramatically faster than hand-writing forty import blocks.

A clean terraform validate is not safety. The one I would underline. The gap between "valid configuration" and "safe plan" is exactly where brownfield imports go wrong, and exactly the gap policy as code fills.

Attach policies before the first import, not after the first scare. Mine were in place before any resource entered state, which is why the EKS near-miss is an anecdote rather than an incident report.

Use both enforcement levels. Hard for invariants that must never break, soft for things a human must consciously approve. One without the other gives you either a workflow you cannot complete or a warning nobody reads.

Import in layers. Smaller plans get read properly, dependencies resolve in order, failures stay localised. The extra runs cost minutes.

Closing thought

The capability to import brownfield infrastructure has existed for years. What has been missing is not capability but economics and confidence.

Search changes the economics — discovery becomes declarative and generation becomes bulk, so adopting forty resources is an afternoon rather than a sprint. Policy as code changes the confidence — you no longer have to be certain you read the plan correctly, because a machine checks the one invariant that matters on every run. MCP-grounded AI assistance makes the authoring in between fast without making it unreliable.

The infrastructure that most needs to be in code is the infrastructure people are most afraid to touch. That fear is rational. It is also, I think, now addressable.

Try it yourself

Demo repo (CloudFormation template, query file, prompts, app): github.com/pranit-hashi/aws-brownfield-terraform-import
Sentinel policies used as guardrails: github.com/pranit-hashi/import-sentinel-policies
HashiCorp Agent Skills for Terraform — highly recommended reading before you use them: github.com/hashicorp/agent-skills/tree/main/plugins/terraform
Terraform MCP Server: developer.hashicorp.com/terraform/mcp-server
Which resources support list/import? An auto-updating report that tracks provider-level support, and the first thing I check before planning an import: list-resources-report.json

Setup used: Terraform 1.14+, hashicorp/aws 6.67.0 and hashicorp/awscc 1.104.0, HCP Terraform with Sentinel policy sets, the Terraform MCP Server, and the terraform-search-import Agent Skill.

If you try this on your own brownfield app, I would like to hear which resource type caught you out. My money is on something in IAM.

How to create a CodePipeline with source from another AWS account?

Pranit Raje — Sun, 22 Mar 2020 15:22:30 +0000

When using AWS services, more often than not, we need multiple AWS accounts for various reasons. If you work on AWS DevOps tools, you must have come across AWS CodePipeline which is a pipeline automation tool from AWS. While working with CodePipeline in multi-account scenario, I'm sure you all have faced many issues like Access Denied errors which is the most puzzling and frustrating error IMHO because it doesn't always say where we're lacking permissions. In multi-account scenario for CodePipeline, if you're deploying your application in another AWS account, AWS has this excellent documentation which I'd highly recommend you to go through. But what if you have a unique requirement when the source of your pipeline itself is in another AWS account and the deployment is in the same account as of CodePipeline?

Recently, I came across this use-case and tried to find some starting guide to create a CodePipeline having source from another AWS account but couldn't find any useful resources. Hence, I decided to write this blog to help others who might run into such use-case.

This blog will walk you through an example of deploying from Amazon ECR to Amazon ECS with the help of CodePipeline. Here, I'm having a use-case in which,

Account-A has already-built Docker image hosted in ECR (Source stage)
Account-B has S3 bucket (Source stage) containing imagedefinitions.json and ECS (Deploy stage).

NOTE: All AWS resources in this example exist in the same AWS region.

Our requirement is such that we have a team of developers in Account-A who will build their Docker images locally, push it in ECR repository in Account-A and this Docker image should be deployed automatically to the resources (ECS) present in Account-B whenever our developers make changes to their Docker images and push it in ECR repository of Account-A. And, we want our CodePipeline to be set up in Account-B itself where our deployment is done. Sounds challenging right? No worries. Let's get started...

Prerequisites:

Two active AWS accounts.
A working Amazon ECS cluster up and running with all other related resources e.g., Task Definition, ECR repository, ECS Service, any Docker image in one of the AWS account.

Target Architecture:

The above diagram shows the following workflow:

Initially, the developers will build the Docker image and push it to their central ECR repository in Account A.
Account A will have a EventBridge Rule created listening to our ECR repository in Account A. As soon as a successful Docker image is pushed to our ECR repository, an EventBridge Rule will be triggered in Account A.
Default EventBridge Bus in Account B will be configured as a target to our EventBridge rule in Account A.
We will have another EventBridge rule created in Account B which will listen to the push events from the ECR repository in Account A.
The EventBridge rule in Account B will trigger our CodePipeline.
CodePipeline will assume a cross-account IAM role created in Account A to list and describe the latest Docker image pushed into ECR repository.
CodePipeline will read the Docker image and pass it to the next stage of CodePipeline.
The imagedefinitions.zip file from S3 bucket will contain the information about the container name and the image URI of our Docker image stored in Account A’s ECR repository which will be used in ECS Standard deployment.

Now, let's see how we can build such pipeline from AWS Management Console in a step-by-step manner,

Step 1: IN ACCOUNT-B -> Set up a normal CodePipeline:

Firstly, create a standard pipeline having all the components of pipeline (ECR,S3,ECS) residing in same Account-B normally the way you would. We are creating this pipeline in the same account first just to get a backbone structure for our cross-account pipeline which we will modify according to our requirements in later steps.
In this pipeline, Source stage will have two actions i.e. ECR and S3 where ECR will be your repository with image tag and S3 bucket will be having zip file of the JSON file named imagedefinitions.json. This JSON file is required for the ECS deployment as it contains container name and the ECR image URI with its image tag. Refer to this official documentation to know more about this file. Do remember to zip the file when S3 is used in the source stage.

NOTE: This S3 bucket should be versioned.
When you include ECS (Standard deployment) as a deployment provider in your pipeline, make sure to point S3 action's output as an input to ECS because ECR action's output will contain information only about the Docker image and not the container name which is expected by CodePipeline job worker for ECS.
Once you have working pipeline in Account-B, we will modify it for cross-account deployments having source from Account-A.

Step 2: IN ACCOUNT-B -> Create custom AWS KMS key:

For any cross-account deployments, CodePipeline require custom KMS key allowing another AWS account to access encrypted CodePipeline artifacts stored in CodePipeline’s S3 bucket (artifact store).
Create custom AWS KMS key allowing Account-A to access CodePipeline artifacts residing in Account-B’s S3 bucket.

While creating KMS policy via console, when you reach to the 5th step Review and edit key policy, enter the below policy by replacing appropriate ARN at the commented places.

{
    "Id": "key-consolepolicy",
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Enable IAM User Permissions",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::<ACCOUNT-B_ID>:root"  //MAKE CHANGES HERE
            },
            "Action": "kms:*",
            "Resource": "*"
        },
        {
            "Sid": "Allow access for Key Administrators",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::<ACCOUNT-B_ID>:user/<YOUR_IAM_USER>"   //MAKE CHANGES HERE
            },
           "Action": [
                "kms:Create*",
                "kms:Describe*",
                "kms:Enable*",
                "kms:List*",
                "kms:Put*",
                "kms:Update*",
                "kms:Revoke*",
                "kms:Disable*",
                "kms:Get*",
                "kms:Delete*",
                "kms:TagResource",
                "kms:UntagResource",
                "kms:ScheduleKeyDeletion",
                "kms:CancelKeyDeletion"
            ],
            "Resource": "*"
        },
        {
            "Sid": "Allow use of the key",
            "Effect": "Allow",
            "Principal": {
                "AWS": [
                    "arn:aws:iam::<ACCOUNT-B_ID>:user/<YOUR_IAM_USER>",   //MAKE CHANGES HERE
                    "arn:aws:iam::<ACCOUNT-B_ID>:role/service-role/AWSCodePipelineServiceRole-<AWS_REGION>-<CODEPIPELINE_NAME>",  //REPLACE WITH CODEPIPELINE SERVICE ROLE ARN
                    "arn:aws:iam::<ACCOUNT-A_ID>:root"  //MAKE CHANGES HERE
                ]
            },
            "Action": [
                "kms:Encrypt",
                "kms:Decrypt",
                "kms:ReEncrypt*",
                "kms:GenerateDataKey*",
                "kms:DescribeKey"
            ],
            "Resource": "*"
        },
        {
            "Sid": "Allow attachment of persistent resources",
            "Effect": "Allow",
            "Principal": {
                "AWS": [
                    "arn:aws:iam::<ACCOUNT-B_ID>:user/<YOUR_IAM_USER>",   //MAKE CHANGES HERE
                    "arn:aws:iam::<ACCOUNT-B_ID>:role/service-role/AWSCodePipelineServiceRole-<AWS_REGION>-<CODEPIPELINE_NAME>",  //REPLACE WITH CODEPIPELINE SERVICE ROLE ARN
                    "arn:aws:iam::<ACCOUNT-A_ID>:root"  //MAKE CHANGES HERE
                ]
            },
            "Action": [
                "kms:CreateGrant",
                "kms:ListGrants",
                "kms:RevokeGrant"
            ],
            "Resource": "*",
            "Condition": {
                "Bool": {
                    "kms:GrantIsForAWSResource": "true"
                }
            }
        }
    ]
}

The above KMS key policy will allow your IAM user, CodePipeline service role and Account-A to access this KMS key with which our artifacts will be encrypted.

Step 3: IN ACCOUNT-A -> Create a custom IAM role for cross-account access:

Create a new IAM role named CrossAccount-A_Role. You can name this role anything you want as long as it follows the naming conventions in IAM. Consider giving the role a name that clearly states its purpose.
On the Create role page, choose Another AWS account option from Select type of trusted entity section. Enter the AWS Account ID of Account-B in the Specify accounts that can use this role section. Click Next: Permissions.
Attach two AWS managed policies named AmazonEC2ContainerRegistryReadOnly and AmazonS3FullAccess to this IAM Role.
Create another custom inline IAM policy named cross-account-KMS for this same IAM role to access Account-B’s KMS key which we created in Step 2. Enter the below IAM policy by replacing ARN of KMS key in Account-B at the commented place for this inline policy.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "kms:DescribeKey",
                "kms:GenerateDataKey*",
                "kms:Encrypt",
                "kms:ReEncrypt*",
                "kms:Decrypt"
            ],
            "Resource": [
                "<ARN_OF_KMS_KEY_IN_ACCOUNT-B>"  //MAKE CHANGE HERE
            ]
        }
    ]
}

Step 4: IN ACCOUNT-B -> Create a policy for the S3 bucket that grants access to Account A:

Choose your CodePipeline’s S3 bucket i.e. artifact store (not the source S3 bucket) and edit its Bucket policy according to the following policy by replacing appropriate ARN at the commented places.
The below S3 bucket policy will allow Account-A to access our pipeline's artifacts.

{
    "Version": "2012-10-17",
    "Id": "SSEAndSSLPolicy",
    "Statement": [
    {
        "Sid": "DenyUnEncryptedObjectUploads",
        "Effect": "Deny",
        "Principal": "*",
        "Action": "s3:PutObject",
        "Resource": "arn:aws:s3:::<CODEPIPELINE'S_S3_BUCKET_NAME>/*",   //MAKE CHANGE HERE
        "Condition": {
                "StringNotEquals": {
                    "s3:x-amz-server-side-encryption": "aws:kms"
                }
            }
    },
    {
        "Sid": "DenyInsecureConnections",
        "Effect": "Deny",
        "Principal": "*",
        "Action": "s3:*",
        "Resource": "arn:aws:s3:::<CODEPIPELINE'S_S3_BUCKET_NAME>/*",   //MAKE CHANGE HERE
        "Condition": {
            "Bool": {
                    "aws:SecureTransport": false
                }
            }
    },
    {
        "Sid": "",
        "Effect": "Allow",
        "Principal": {
            "AWS": "arn:aws:iam::<ACCOUNT-A_ID>:root"   //MAKE CHANGE HERE
            },
        "Action": [
                "s3:Get*",
                "s3:Put*"
            ],
        "Resource": "arn:aws:s3:::<CODEPIPELINE'S_S3_BUCKET_NAME>/*"   //MAKE CHANGE HERE
    },
    {
        "Sid": "",
        "Effect": "Allow",
        "Principal": {
            "AWS": "arn:aws:iam::<ACCOUNT-A_ID>:root"   //MAKE CHANGE HERE
                },
        "Action": "s3:ListBucket",
        "Resource": "arn:aws:s3:::<CODEPIPELINE'S_S3_BUCKET_NAME>"   //MAKE CHANGE HERE
        }
    ]
}

Step 5: IN ACCOUNT-B -> Create an inline policy for service role of CodePipeline:

As we want CodePipeline service role to assume the cross-account IAM role we created in Step 3, we should add an extra inline IAM policy to the CodePipeline service role.
Enter the below IAM policy by replacing Account-A ID at the commented place for this inline policy.

{
    "Version": "2012-10-17",
    "Statement": {
        "Effect": "Allow",
        "Action": "sts:AssumeRole",
        "Resource": [
            "arn:aws:iam::<ACCOUNT-A_ID>:role/*"   //MAKE CHANGE HERE
        ]
    }
}

Step 6: IN ACCOUNT-A -> Apply ECR Repository Policy:

To allow CodePipeline in Account-B to pull ECR images residing in Account-A, ECR repository should allow Account-B to pull those images from its repository. For this, we will apply resource-based policy to our ECR repository.
You can use the below ECR repository policy by replacing appropriate ARN at the commented places.

{
  "Version": "2008-10-17",
  "Statement": [
    {
      "Sid": "Cross-account-policy",
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "arn:aws:iam::<ACCOUNT-B_ID>:root",   //MAKE CHANGE HERE
          "arn:aws:iam::<ACCOUNT-A_ID>:role/CrossAccount-A_Role"   //ARN OF CROSS-ACCOUNT IAM ROLE CREATED IN ACCOUNT-A AT STEP 3   
        ]
      },
      "Action": [
        "ecr:BatchCheckLayerAvailability",
        "ecr:BatchGetImage",
        "ecr:CompleteLayerUpload",
        "ecr:DescribeImages",
        "ecr:DescribeRepositories",
        "ecr:GetDownloadUrlForLayer",
        "ecr:GetLifecyclePolicy",
        "ecr:GetLifecyclePolicyPreview",
        "ecr:GetRepositoryPolicy",
        "ecr:ListImages",
        "ecr:UploadLayerPart"
      ]
    }
  ]
}

Step 7: IN ACCOUNT-B -> Modify your pipeline’s configuration file:

Now that we have most of the required components of our cross-account pipeline ready, we will modify the configuration file of our pipeline that we created in Step 1 by changing the source to our ECR repository in Account-A and we will also add KMS related details.
To do that, run get-pipeline command aws codepipeline get-pipeline --name <CODEPIPELINE_NAME> --region <AWS_REGION> > pipeline.json from AWS CLI.

Open this pipeline.json file in your favourite text editor (vim?) and add ‘encryptionKey’ in ‘artifactStore’ section.

 "artifactStore": {
    "type": "S3",
    "location": "codepipeline-<AWS_REGION>-XXXXXXXXXX",  //MAKE CHANGE HERE
    "encryptionKey": {
        "id": "arn:aws:kms:<AWS_REGION>:<ACCOUNT-B_ID>:key/XXXXXXXXXX-XXXX-XXX-XXXX-XXXXXXXXXX",    //MAKE CHANGE HERE    
        "type": "KMS"
        }
     }

Then, modify your ECR action of Source stage where you will change the values of ECR repository name and its image tag which is in Account-A.

In the same ECR action, include the "roleArn": "arn:aws:iam::<ACCOUNT-A_ID>:role/CrossAccount-A_Role" which we created in Step 3 so that this cross-account IAM role from Account-A will be used for ECR action. Following is the snippet for our cross-account source action in pipeline's JSON file.

NOTE: Replace appropriate values at commented places below

{
    "name": "Source",
    "actionTypeId": {
      "category": "Source",
      "owner": "AWS",
      "provider": "ECR",
      "version": "1"
     },
    "runOrder": 1,
    "configuration": {
      "ImageTag": "latest",    //MAKE CHANGE HERE IN CASE OF CUSTOM TAG
      "RepositoryName": "XXXXX"    //MAKE CHANGE HERE
     },
    "outputArtifacts": [
      {
        "name": "SourceArtifact"
      }
     ],
    "inputArtifacts": [],
    "region": "<AWS_REGION>",  //MAKE CHANGE HERE
    "namespace": "SourceVariables",
    "roleArn": "arn:aws:iam::<ACCOUNT-A_ID>:role/CrossAccount-A_Role"  //MENTION CROSS-ACCOUNT IAM ROLE    
}

NOTE: Above is just a snippet of our ECR action of Source stage and not the entire JSON configuration of pipeline.

Make sure to perform the 4th step of this documentation to remove the unwanted metadata section from configuration file.
Once you make the above changes, update the pipeline from update-pipeline CLI command aws codepipeline update-pipeline --region <AWS_REGION> --cli-input-json file://pipeline.json

After you update your pipeline, don't forget to update your imagedefintions.json file with the correct imageURI from Account-A if you haven't done so. It should look like below,

[
  {
    "name": "<CONTAINER_NAME>",  //MAKE CHANGES HERE
    "imageUri": "<ACCOUNT-A_ID>.dkr.ecr.<AWS_REGION>.amazonaws.com/<REPOSITORY_NAME>:<IMAGE_TAG>"  //MAKE CHANGES HERE    
  }
]

Once updated the imagedefinitions.json file, zip it and upload it again to your source S3 bucket.

As you might be aware that CloudWatch Event (CWE) Rules are the resources in the background which trigger your pipeline whenever there are any changes to your source present in AWS. Since, one of our source action, i.e. ECR is existing in another AWS account, CodePipeline cannot create this CWE rule automatically for you for ECR as it did for S3 action. Hence, we will now create CWE rule for ECR repository with the help of CloudWatch Event Bus.

Step 8: IN ACCOUNT-B -> Edit default Event Bus to allow account A:

Go to CloudWatch console and select your default Event Bus.
Click on Add permission to add your Account-A ID and check the Everybody(*) option.

Step 9: IN ACCOUNT-A -> Create CWE Rule for your ECR repository:

Follow this official documentation till Step 5 to create CWE rule for your ECR repository as a source for your CWE rule.
As a target for this CWE rule, select Event bus in another AWS account. Enter your Account-B ID and select Create a new role for this specific resource option for IAM role which will automatically create IAM role for this CWE rule.

Step 10: IN ACCOUNT-B -> Create CWE Rule with CodePipeline as a target:

Create a new CWE rule with Event Source exactly same as you made in Account-A. Edit this event pattern by adding account section with the value as Account-A’s ID. This will listen to any changes occurring to your ECR repository present in Account-A with the help of Event Bus. Refer to this official documentation to know how to do this exactly. Once you create Event pattern, it should look like below,

{
  "detail-type": [
    "ECR Image Action"
  ],
  "account": [
    "ACCOUNT-A_ID"  //MAKE CHANGES HERE
  ],
  "source": [
    "aws.ecr"
  ],
  "detail": {
    "action-type": [
      "PUSH"
    ],
    "image-tag": [
      "latest"  //MAKE CHANGE HERE IN CASE OF CUSTOM TAG
    ],
    "repository-name": [
      "<REPOSITORY_NAME>"  //MAKE CHANGES HERE
    ],
    "result": [
      "SUCCESS"
    ]
  }
}

Put your CodePipeline ARN (arn:aws:codepipeline:<AWS_REGION>:<ACCOUNT-B_ID>:<PIPELINE_NAME>) as a target for this CWE rule.

Hooray!! Now, we have successfully set up CodePipeline in Account-B with ECR being in Account-A, S3 bucket containing only zipped imagedefinitions.json file for ECS deployment and ECS service in Account-B. Whenever you make any changes to any of your source, your pipeline will be triggered in a CI/CD manner.

Automation & Scale:

Following the step-by-step instructions from the above section can be useful if we want to understand what we are doing to implement the architecture. However, once we understand that and when we want to automate the whole target architecture, it can be deployed in an automated way using AWS CloudFormation which is an Infrastructure as Code (IaC) tool from AWS. To automate the deployment in multiple AWS accounts, make sure to use the CloudFormation templates from GitHub repository. In the repository, we have three CloudFormation templates which should be deployed sequentially as noted below:

In Account B, create a new CloudFormation stack using AccB-pipeline-1.yaml template.
In Account A, create a new CloudFormation stack using AccA-pipeline.yaml template.
In Account B, run following AWS CLI command from Account B to allow Account A to send ECR push events to Event Bus in Account B. aws events put-permission --action events:PutEvents --statement-id MySid --principal <ACCOUNT_A_ID>
In Account B, update the CloudFormation stack created in Step 1 using AccB-pipeline-2.yaml template. In this template, we will use one additional parameter named CrossAccountRoleName which will be the name of cross-account role created in Step 2. The IAM role name will be available from the Outputs section of CloudFormation console. Along with that, we will also put the ECR repository name created in Account A in Step 2 which is available in the Outputs section of CloudFormation console.

In Account B, make sure to update imagedefintions.json file with the correct imageURI from Account A. It may look like below,

[
{
"name": "<CONTAINER_NAME>",  //MAKE CHANGES HERE
"imageUri": "<ACCOUNT-A_ID>.dkr.ecr.<AWS_REGION>.amazonaws.com/<REPOSITORY_NAME>:<IMAGE_TAG>"  //MAKE CHANGES HERE    
}
]

In Account A, push already built Docker image to ECR repository in Account A.
In Account B, notice how CodePipeline will get triggered automatically as soon as any Docker image is pushed to ECR repository in Account A. This Docker image will be deployed to ECS cluster in Account B in CI/CD manner.

Thanks for reading this blog! I hope you will find this useful. Feel free to like, share and comment on this article with your constructive feedback. If you find this blog helpful or if you're stuck at any of the above step, do let me know via comments section and I'd be more than happy to help!

Have an AWSome Day! 😎