Introduction:
In this blog post, we have the process of modeling GitOps environments and promoting releases between them using AWS CloudFormation’s Git Sync feature. Here we use the “environment-per-folder” approach, more information on how to model GitOps environments and promote releases between them is in this article. More information about the CloudFormation Git Sync concept, prerequisites, and walkthrough is here.
About the project:
In this project we have 2 folders. The folder git_connection has CloudFormation template git_connection.yaml which creates all necessary configurations for the creation of CloudFormation stacks with the Git Sync option. With this stack, we create resources: CodeStar connection for connection to our Gitlab account, two IAM roles — first to update the stack from the Git repository and second to use for all operations performed on the stack, SSM parameter for S3 bucket prefix name. The folder infrastructure has environment folders. In every folder, we have a deployment file and a template file. In the template file, we have a simple configuration for the creation of an S3 bucket with an S3 Bucket policy. We assume that we have the same template file for all environments and specify parameters in our deployment files.
The project structure:
├── git_connection
│ └── git_connection.yaml
└── infrastructure
├── development
│ ├── deployment_parameters.yaml
│ └── s3bucket.yaml
├── production
│ ├── deployment_parameters.yaml
│ └── s3bucket.yaml
└── staging
├── deployment_parameters.yaml
└── s3bucket.yaml
The CloudFormation git_connection.yaml template:
AWSTemplateFormatVersion: '2010-09-09'
Description: 'Connect Gitbab repository with AWS'
Parameters:
ConnectionName:
Type: String
Default: 'Gitlab-to-CloudFormation'
S3BucketPrefixName:
Type: String
Default: 'cf-app-files'
Resources:
GitLabConnection:
Type: 'AWS::CodeStarConnections::Connection'
Properties:
ConnectionName: !Ref ConnectionName
ProviderType: 'GitLab'
CloudFormationS3AccessRole:
Type: AWS::IAM::Role
Properties:
RoleName: CloudFormationS3AccessRole
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
Service:
- cloudformation.amazonaws.com
Action:
- sts:AssumeRole
Policies:
- PolicyName: CloudFormationS3ManagementPolicy
PolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Action:
- s3:CreateBucket
- s3:DeleteBucket
- s3:PutBucketPolicy
- s3:GetBucketPolicy
- s3:ListBucket
- s3:PutBucketTagging
- s3:PutBucketPolicy
- s3:DeleteBucketPolicy
Resource: '*'
- Effect: Allow
Action:
- ssm:GetParameters
Resource: '*'
- Effect: Allow
Action:
- cloudformation:*
Resource: '*'
GitSyncRole:
Type: AWS::IAM::Role
Properties:
RoleName: CFNGitSyncRole
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Sid: CfnGitSyncTrustPolicy
Effect: Allow
Principal:
Service: cloudformation.sync.codeconnections.amazonaws.com
Action: sts:AssumeRole
Policies:
- PolicyName: GitSyncPermissions
PolicyDocument:
Version: 2012-10-17
Statement:
- Sid: SyncToCloudFormation
Effect: Allow
Action:
- cloudformation:CreateChangeSet
- cloudformation:DeleteChangeSet
- cloudformation:DescribeChangeSet
- cloudformation:DescribeStackEvents
- cloudformation:DescribeStacks
- cloudformation:ExecuteChangeSet
- cloudformation:GetTemplate
- cloudformation:ListChangeSets
- cloudformation:ListStacks
- cloudformation:ValidateTemplate
Resource: '*'
- Sid: PolicyForManagedRules
Effect: Allow
Action:
- events:PutRule
- events:PutTargets
Resource: '*'
Condition:
StringEquals:
events:ManagedBy: cloudformation.sync.codeconnections.amazonaws.com
- Sid: PolicyForDescribingRule
Effect: Allow
Action: events:DescribeRule
Resource: '*'
SsmS3BucketPrefixName:
Type: AWS::SSM::Parameter
Properties:
Name: S3BucketPrefixName
Type: String
Value: !Ref S3BucketPrefixName
Description: "Prefix of the S3 bucket name for git sync cf stacks"
The CloudFormation s3bucket.yaml template:
AWSTemplateFormatVersion: '2010-09-09'
Description: 'S3 bucket with a dynamically created name and apply a policy for access from EC2 instances'
Parameters:
S3BucketPrefixName:
Type: AWS::SSM::Parameter::Value<String>
Default: ''
Environment:
Description: List of available environments
Type: String
Default: dev
AllowedValues:
- dev
- stag
- prod
ConstraintDescription: Use valid environment [dev, stag, prod]
Mappings:
EnvironmentLabel:
dev:
label: development
stag:
label: staging
prod:
label: production
Resources:
S3BucketFiles:
Type: AWS::S3::Bucket
Properties:
BucketName:
!Sub
- '${S3BucketPrefixName}-${UsedEnvironmentLabel}'
- UsedEnvironmentLabel: !FindInMap [ EnvironmentLabel, !Ref Environment, label ]
Tags:
- Key: Name
Value:
!Sub
- '${S3BucketPrefixName}-${UsedEnvironmentLabel}'
- UsedEnvironmentLabel: !FindInMap [ EnvironmentLabel, !Ref Environment, label ]
- Key: Environment
Value: !FindInMap [ EnvironmentLabel, !Ref Environment, label ]
S3BucketPolicy:
Type: AWS::S3::BucketPolicy
Properties:
Bucket: !Ref S3BucketFiles
PolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
Service:
- ec2.amazonaws.com
Action:
- s3:GetObject
Resource:
!Sub
- 'arn:${AWS::Partition}:s3:::${S3BucketPrefixName}-${UsedEnvironmentLabel}/*'
- UsedEnvironmentLabel: !FindInMap [ EnvironmentLabel, !Ref Environment, label ]
The CloudFormation development/deployment_parameters.yaml template:
template-file-path: './infrastructure/development/s3bucket.yaml'
parameters:
S3BucketPrefixName: 'S3BucketPrefixName'
Environment: 'dev'
tags:
InfraDeploymentProcess: 'cf stack with git sync option for dev env'
Prerequisites:
Before you start, make sure the following requirements are met:
- An AWS account with permissions to create resources.
- AWS CLI installed on your local machine.
- Account in any Git provider for remote Git repository.
Deployment:
- Clone the repository
git clone https://gitlab.com/Andr1500/cloudformation_sync_from_git.git
- Creation of necessary configuration for interconnection between Gitlab and AWS account.
a. Create CloudFormation stack with CodeStar connection and necessary IAM roles:
aws cloudformation create-stack \
--stack-name cf-git-sync-config \
--template-body file://git_connection/git_connection.yaml \
--capabilities CAPABILITY_NAMED_IAM
b. Make necessary changes, commit it, and push it to your remote repository.
c. Update CodeStar pending connection. Open the AWS Console, next go: CodePipeline -> Settings -> Connections -> choose the created connection in pending status -> Update pending connection -> depends on your provider make authorisation, and grant necessary access to the repository.
- PreBuild step (optional) before CloudFormation Git Sync stack creation.
Sometimes, if you already earlier linked the repository with the AWS account by CodeStar connection after deleting it and recreating again, you will have the issue during the process of creating the CloudFormation stack: the repository will be already linked to the “old” connection. In this case, we should “unlink” the repository with the AWS CLI command by deleting “old” repository link and “link” again:
aws codestar-connections list-repository-links
aws codestar-connections delete-repository-link --repository-link-id 1234567-76a3-4f20-858f-qwerty12345
- Creation of CloudFormation stacks for different environments.
Open the AWS Console, next go: CloudFormation -> Create stack (button) -> With new resources -> Choose options “Template is ready” and “Sync from Git” -> Next -> Provide stack name (for example, “cf-git-deployment”), choose option “I am providing my own file in my repository”, choose option “Link a Git repository” (if you create it for the first time), choose the repository provider, connection, repository, branch, provide the deployment file path, choose the IAM role for CloudFormation to update the stack from the Git repository -> Next -> choose the IAM role for CloudFormation to use for all operations performed on the stack, choose the stack failure options -> Next -> Review and create -> Submit.
Repeat a similar procedure for creating CloudFormation stacks for other environments, in this case, we need to specify different CF stack names and different deployment file paths.
- Update CloudFormation stacks and promote changes between the environments.
IMPORTANT ! All changes (Creation of new resources, updating existing resources) with the created CloudFormation stacks should be done with the Git repository, not with the CloudFormation “Update” button.
a. Make necessary changes in the CF template file for dev env, commit changes, and push it. Check if the changes applied in the CloudFormation stack are related to the development environment.
b. Make a copy of the template file from one environment to another (for example, from dev env to stag env), commit changes, and push it. Check if the changes applied in the CloudFormation stack are related to a staging environment.
cp infrastructure/development/s3bucket.yaml infrastructure/staging/s3bucket.yaml
Conclusion:
imho, for now, the CloudFormation Git Sync feature has some advantages and disadvantages. As one of the advantages — we have only one source of code and we can simply manage changes in our infrastructure. As a disadvantage — we can’t create nested stack with this feature because in AWS::CloudFormation::Stack resource TemplateURL parameter supports only the S3 bucket URL. Also, we can’t integrate the created Webhook into our Gitlab CI/CD pipeline because the Webhook’s secret token is hidden and we don’t have any option for using existed Webhook in our pipeline. We can’t create CloudFormation Git Sync stack with the AWS CLI command create-stack because, similar to nested stack, — template-url supports only the S3 bucket URL.
If you found this post helpful and interesting, please click the clap button below to show your support. Feel free to use and share it.
Top comments (0)