How to sync CloudFormation templates from CodeCommit to S3

If it happens to you to work on a project where you have selected for one or another reason CloudFormation, you most probably have fallen into a situation to ask yourself the same question as me "OK! But how could I reduce the manual effort and risk from human error while maintaining my CloudFormation Templates between CodeCommit and S3?".
You can easily replace CodeCommit with any other SCM platform, however, this topic will be still valid.

CloudFormation is the managed service offered by AWS for managing your infrastructure as a code. Whether you prefer JSON or YAML files you will always need a location to centrally store your templates.

Use case

Keep track of the changes made to the files when in the meantime you do not need to sync templates to the S3 bucket artifact location manually which will increase the risk of human error during the manual synchronization.

Question

How can we automate this and always have near-real-time synchronization between our repo and S3 bucket from which we point URLs inside the CloudFormation stacks?

Solution

As with every topic we need to cover, the first thing to do is what?
Diagram of course ...

Here is a sample of a CloudFormation template that we can utilize to achieve the goal.
A prerequisite is to have a CodeCommit repository created in advance.

AWSTemplateFormatVersion: 2010-09-09
Parameters:
  pCodeCommitRepo:
    Description: This parameter serves to point which is our repository to use as a single point of truth
  pSolutionName:
    Description: Name which we can use to spin up more than one Pipeline for different repositories
  pBucketName:
    Type: String
    Default: my-cf-templates
  pMyAWSKeyAccountId:
    Type: String
    Default: 123456789012
  pMyAWSKeyRegion:
    Type: String
    Default: eu-west-1
  pMyAWSKeyArn:
    Type: String
    Default: arn:aws:kms:eu-west-1:123456789012:key/abcdefgh-ijkl-1234-5678-zxcvbnmasdfg

Resources:
  rS3Bucket:
    Type: AWS::S3::Bucket
    DeletionPolicy: Retain
    Properties:
      BucketName: !Ref pBucketName
      VersioningConfiguration:
        Status: Enabled
      PublicAccessBlockConfiguration:
          BlockPublicAcls: true
          BlockPublicPolicy: true
          IgnorePublicAcls: true
          RestrictPublicBuckets: true
      BucketEncryption:
        ServerSideEncryptionConfiguration:
          - ServerSideEncryptionByDefault:
              SSEAlgorithm: 'aws:kms'
              KMSMasterKeyID: !GetAtt rKMSS3Encryption.Arn
      # What we can do here, if we have multiple AWS accounts where we are going to use the CloudFormation Templates is to replicate
      # the content in a dedicated S3 bucket inside every AWS account where we need.
      ReplicationConfiguration:
          Role: !GetAtt rS3ReplicationRole.Arn
          Rules:
            - Destination:
                Bucket: !Sub "arn:aws:s3:::my-cf-templates-${pMyAWSKeyAccountId}.${pMyAWSKeyRegion}"
                AccessControlTranslation:
                  Owner: Destination
                Account: !Ref pMyAWSKeyAccountId
                EncryptionConfiguration:
                  ReplicaKmsKeyID: !Ref pMyAWSKeyArn
              DeleteMarkerReplication:
                Status: Enabled
              Priority: 1
              Filter:
                Prefix: ''
              Status: Enabled
              SourceSelectionCriteria:
                SseKmsEncryptedObjects:
                  Status: Enabled

  rKMSS3Encryption:
    Type: AWS::KMS::Key
    DeletionPolicy: Retain
    Properties:
      Enabled: true
      KeyPolicy:
        Version: '2012-10-17'
        Id: key-default-1
        Statement:
          - Sid: Enable IAM User Permissions
            Effect: Allow
            Principal:
              AWS:
                Fn::Join:
                - ''
                - - 'arn:aws:iam::'
                  - Ref: AWS::AccountId
                  - :root
            Action: kms:*
            Resource: '*'
          - Sid: Allow  use of the CMK
            Effect: Allow
            Principal:
              AWS:
                - !Sub arn:aws:iam::${AWS::AccountId}:role/service-role/AWSCodePipelineServiceRole-*
            Action:
                - kms:Encrypt
                - kms:Decrypt
                - kms:ReEncrypt*
                - kms:GenerateDataKey*
                - kms:DescribeKey
            Resource: "*"

      EnableKeyRotation: true

  rKMSS3EncryptionAlias:
      Type: AWS::KMS::Alias
      DeletionPolicy: Retain
      Properties:
        AliasName: alias/myS3BucketKey
        TargetKeyId: !Ref rKMSS3Encryption

  rS3ReplicationPolicy:
    Type: AWS::IAM::ManagedPolicy
    Properties:
      ManagedPolicyName: !Sub myReplicationPolicy-${pBucketName}-${AWS::Region}
      Description: S3 replication policy
      PolicyDocument:
        Version: '2012-10-17'
        Statement:
          -
            Effect: Allow
            Action:
              - 's3:GetReplicationConfiguration'
              - 's3:ListBucket'
            Resource: !Sub arn:aws:s3:::${pBucketName}
          -
            Effect: Allow
            Action:
              - 's3:GetObject*'
            Resource: !Sub arn:aws:s3:::${pBucketName}/*
          -
            Effect: Allow
            Action:
              - 's3:ReplicateObject'
              - 's3:ReplicateDelete'
              - 's3:ReplicateTags'
              - 's3:ObjectOwnerOverrideToBucketOwner'
            Resource: !Sub arn:aws:s3:::my-cf-templates-${pMyAWSKeyAccountId}.${pMyAWSKeyRegion}/*
          -
            Effect: Allow
            Action:
              - 'kms:*'
            Resource: !GetAtt rKMSS3Encryption.Arn
          -
            Effect: Allow
            Action:
              - 'kms:*'
            Resource: !Ref pMyAWSKeyArn

  rS3ReplicationRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: !Sub ReplicationRole-${pBucketName}-${AWS::Region}
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: 'Allow'
            Principal:
              Service:
                - 's3.amazonaws.com'
            Action:
              - 'sts:AssumeRole'
      ManagedPolicyArns:
        - !Ref rS3ReplicationPolicy

  rCodePipeline:
    Type: AWS::CodePipeline::Pipeline
    Properties:
      Name: !Sub "${pSolutionName}-pipeline"
      RestartExecutionOnUpdate: false
      RoleArn: !GetAtt rCodePipelineServiceRole.Arn
      Stages:
        - Name: Source
          Actions:
            - Name: SourceAction
              ActionTypeId:
                Category: Source
                Owner: AWS
                Provider: CodeCommit
                Version: 1
              Configuration:
                RepositoryName: !Ref pCodeCommitRepo
                BranchName: master
              RunOrder: 1
        - Name: SyncToS3
          Actions:
            - Name: SyncToS3
              ActionTypeId:
                Category: Deploy
                Owner: AWS
                Version: 1
                Provider: CodeDeploy
              Configuration:
                Extract: true
                BucketName: !Ref rS3Bucket
                ObjectKey: "/"
              RunOrder: 1
  rCodePipelineServiceRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Principal:
              Service:
                - codepipeline.amazonaws.com
            Action: 'sts:AssumeRole'
      Path: /
      Policies:
        - PolicyName: !Sub "AWS-CodePipeline-Service-${pCodeCommitRepo}"
          PolicyDocument:
            Version: 2012-10-17
            Statement:
              - Effect: Allow
                Action:
                  - 'codecommit:CancelUploadArchive'
                  - 'codecommit:GetBranch'
                  - 'codecommit:GetCommit'
                  - 'codecommit:GetUploadArchiveStatus'
                  - 'codecommit:UploadArchive'
                Resource: '*'
              - Effect: Allow
                Action:
                  - 'codedeploy:CreateDeployment'
                  - 'codedeploy:GetApplicationRevision'
                  - 'codedeploy:GetDeployment'
                  - 'codedeploy:GetDeploymentConfig'
                  - 'codedeploy:RegisterApplicationRevision'
                Resource: '*'
              - Effect: Allow
                Action:
                  - 'codebuild:BatchGetBuilds'
                  - 'codebuild:StartBuild'
                Resource: '*'
              - Effect: Allow
                Action:
                  - 'devicefarm:ListProjects'
                  - 'devicefarm:ListDevicePools'
                  - 'devicefarm:GetRun'
                  - 'devicefarm:GetUpload'
                  - 'devicefarm:CreateUpload'
                  - 'devicefarm:ScheduleRun'
                Resource: '*'
              - Effect: Allow
                Action:
                  - 'lambda:InvokeFunction'
                  - 'lambda:ListFunctions'
                Resource: '*'
              - Effect: Allow
                Action:
                  - 'iam:PassRole'
                Resource: '*'
              - Effect: Allow
                Action:
                  - 'elasticbeanstalk:*'
                  - 'ec2:*'
                  - 'elasticloadbalancing:*'
                  - 'autoscaling:*'
                  - 'cloudwatch:*'
                  - 's3:*'
                  - 'sns:*'
                  - 'cloudformation:*'
                  - 'rds:*'
                  - 'sqs:*'
                  - 'ecs:*'
                Resource: '*'
  rAmazonCloudWatchEventRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Principal:
              Service:
                - events.amazonaws.com
            Action: 'sts:AssumeRole'
      Path: /
      Policies:
        - PolicyName: !Sub "cw-event-pipeline-execution-${pCodeCommitRepo}"
          PolicyDocument:
            Version: 2012-10-17
            Statement:
              - Effect: Allow
                Action: 'codepipeline:StartPipelineExecution'
                Resource: !Join 
                  - ''
                  - - 'arn:aws:codepipeline:'
                    - !Ref 'AWS::Region'
                    - ':'
                    - !Ref 'AWS::AccountId'
                    - ':'
                    - !Ref rCodePipeline
  rAmazonCloudWatchEventRule:
    Type: AWS::Events::Rule
    Properties:
      EventPattern:
        source:
          - aws.codecommit
        detail-type:
          - CodeCommit Repository State Change
        resources:
          - !Join 
            - ''
            - - 'arn:aws:codecommit:'
              - !Ref 'AWS::Region'
              - ':'
              - !Ref 'AWS::AccountId'
              - ':'
              - !Ref pCodeCommitRepo
        detail:
          event:
            - referenceCreated
            - referenceUpdated
          referenceType:
            - branch
          referenceName:
            - master
      Targets:
        - Arn: !Join 
            - ''
            - - 'arn:aws:codepipeline:'
              - !Ref 'AWS::Region'
              - ':'
              - !Ref 'AWS::AccountId'
              - ':'
              - !Ref rCodePipeline
          RoleArn: !GetAtt rAmazonCloudWatchEventRole.Arn
          Id: codepipeline

In a conclusion, I could say that our target should always be to remove any repetitive effort through automation for the sake of reducing the risk of making errors and at the same time focus on more important and interesting tasks.