IaC management on AWS cloud environment with CDK (pt. 1)

#devops #cdk #aws #python

IaC management on AWS cloud environment with CDK (pt. 1)

Overview

The concept for this series of articles is to explain what and how the CDK framework interacts with AWS to create cloud infrastructure.
The series will be split into three articles:

The first -this one- will introduce the logic of the framework and its steps, starting with getting hands-on with the code.
The second article will address the management of IAM permissions and CI/CD pipelines on Github.
The third article is the implementation of a small project that will cover the use of some AWS services

The language of reference will be Python.

Introduction to aws cdk

Cloud Development Kit (aka CDK) è un framework con il quale si definisce l'infrastruttura che dovrà essere creata su AWS.

The AWS CDK lets you build reliable, scalable, cost-effective applications in the cloud with the considerable expressive power of a programming language.
- Official docs

Currently the supported languages are Typescript, Python, Go, C#, and Java. The definition of the infrastructure then is done through programming languages that allow for greater flexibility and maintainability than Cloudformation, as well as the fact that it is not necessary to learn new semantics, but rather it is enough to know a language and the aws_cdk libraries/API. The output of CDK is a Cloudformation file that can be used to deploy your infrastructure.

CDK is a framework that allows the creation of infrastructure exclusively on AWS, making it locked to a single vendor; however, there are alternatives such as Pulumi or Terraform, which give the ability to interface across multiple providers. What you recommend then is to evaluate well the purpose and goals of your cloud structure. Although I think that multi-cloud makes sense in a few business contexts.

To elaborate further, there are also two other CDK frameworks:

cdk8s that allows manifest definition using programming languages so that .yml or Helm files are avoided
cdktf which wraps terraform allowing infrastructure definition using a language (Still not very mature)
## How it works?

CDK can be divided into three distinct phases, bootstrap, synth, and deployment, let's look at them in detail:

bootstrap is the phase in which the deployed code is executed generating the Cloudformation Toolkit template as output. In practice, it connects to AWS using the profile, account, and region chosen. It will then go on to create a stack that will lay out the basic resources for the CDK environment, i.e., an S3 bucket containing the Cloudformation infrastructure files, IAM roles to run the stacks, SSM parameter for versioning, and ECR.
synth is the phase where the defined code is translated into Cloudformation files, by default it is written to the cdk.out folder.
deploy in this phase the infrastructure stack is created and deployed using the files in cdk.out. The provisioning of required resources may vary by service, in output, all progress is shown

The concept behind CDK is that by using one programming language among those supported it is possible to implement the AWS infrastructure for your services, as seen above it is divided into several phases, and it is possible to manage each phase separately thus having full control over provisioning.
The versatility of CDK allows you to define multiple stacks in the same repository and manage their deployment separately and independently, so there are three key files to consider:

cdk.json is the configuration file where the application to be used when running the CLI cdk commands in the repository and other settings are defined. For a more in-depth.
app.py (the counterpart of cdk.ts in Typescript) is the entry point of our infrastructure in this file the stacks are declared, each stack can be deployed autonomously and independently through the CLI, and it can also contain security logic such as region constraints on which to provisioning.
stack.py (the name may change depending on how you ran the CDK init command) is the file where the infrastructure will be defined using the constructs and classes for the services you want to use

The other concepts regarding cdk's application logic you can learn more about in the official documentation

Review

To recap we have our repository containing configuration files, dependencies, and one or more stack files.
Each stack file defines the services to be created, specifically, each stack class uses modules for each service it is intended to implement. The modules contain the service constructors, which can be of two types:

L1 which is exactly how the resources defined by Cloduformation are.
L2 are constructs that extend the base class Construct and define a set of methods and properties to interact with the resource. You can then manipulate the resource or pass it to other constructs so that you can define connections, example grant permissions.

Hands-on

Our goal is to create the infrastructure with CDK.
You can clone the project related to this series from then
https://github.com/FabrizioCafolla/cdk-demo or follow the following steps by hand:

Install cdk

npm install -g aws-cdk

Create repository

export APPNAME="cdk-demo"
mkdir ${APPNAME}
cd ${APPNAME}

cdk init app --language=python

code . # if use vscode

Now we can implement the stack in cdk_demo/cdk_demo_stack.py (if you used the same APPNAME). Let's create a simple S3 bucket:

import os

from aws_cdk import Stack
from aws_cdk import aws_s3 as s3
from constructs import Construct


class CdkDemoStack(Stack):

    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        self.stage = os.environ['ENVIRONMENT']

        s3.Bucket(
            self,
            'DemoBucket',
            bucket_name=f'demo-{self.environment.account}-{self.stage}-{self.environment.region}',
        )

Now we define the stack in the app.py file

#!/usr/bin/env python3
import os

import aws_cdk as cdk

from cdk_demo.cdk_demo_stack import CdkDemoStack

app = cdk.App()

region = os.getenv('CDK_DEFAULT_REGION')
account_id = os.getenv('CDK_DEFAULT_ACCOUNT')

if region not in ['eu-west-1']:
    raise Exception(f'CDK_DEFAULT_REGION must be eu-west-1, but is: {region}')

if os.getenv('ENVIRONMENT') is None:
    raise Exception(f'var ENVIRONMENT is not set')

environment = cdk.Environment(account=account_id, region=region)

CdkDemoStack(
    app,
    construct_id='CdkDemoStack',
    stack_name='CdkDemoStack',
    description='Demo stack with cdk',
    env=environment,
    synthesizer=cdk.DefaultStackSynthesizer(
        qualifier='demo',
        file_assets_bucket_name=f'{account_id}-cdk-demo-toolkit',
    )
)

app.synth()

In this case we defined constraints on the region, and set the Synthesizer defining which bucket to fetch Cloudformation files from. Of defualt the synthesizer parameter can be non-passed but only in case the --toolkit-bucket-name option was not used at bootstrap, which allows you to choose a name instead of letting it be defined automatically

Now that we have defined the stack and constructs to create an s3 bucket we can deploy the infrastructure

export AWS_PROFILE=name-of-aws-profile      # req. | or export aws credentials
export AWS_ACCOUNT_ID=aws-account-id        # req. 
export AWS_REGION=region-name               # if profile has default region, this line is not needed
export ENVIRONMENT=staging|qa|production    # if you need inside the stack to know the environment  

# Create CdkDemoToolkit stack on Cloudformation
cdk bootstrap --toolkit-stack-name CdkDemoToolkit --toolkit-bucket-name ${AWS_ACCOUNT_ID}-cdk-demo-toolkit --qualifier demo

# Create cdk.out folder with templates
cdk synth CdkDemoStack

# Upload templete on ${AWS_ACCOUNT_ID}-cdk-demo-toolkit bucket and deploy stack 
cdk deploy CdkDemoStack

Delete stack

cdk destroy CdkDemoStack

Notice well how in step 3 resources are defined through L2 constructs (in this case an S3 bucket) while in step 4 stacks are implemented (in this case CdkDemoStack).

Best Practices

There are some of the best practices that should be followed to have more control over the infrastructure:

Separate the logic carefully from the logic of the Constructs from the Stacks, and make the constructs agnostic, so that they can be instantiated in multiple tasks if needed
Separate Stacks only if necessary, it is recommended to separate them in case of stateful stacks and stateless stacks, for example, separate DB or S3 from the application stack.
Assign Tags to created resources, beware that by default CDK adds tags, if they conflict CDK will fail
Try to omit names to resources as much as possible so that they are gender by CDK, this improves management as it avoids name conflicts.
Avoid using Conditions and Parameters constructs, but use the programming language for handling parameters and conditions
Set when needed outputs and possibly export them for use in other stacks
Test the infrastructure using unit tests
Avoid using environment variables in constructs or stacks
Use roles with permissions that are consistent with the infrastructure you create so you don't give too much power during deployment

Considerations

Creating the infrastructure directly on Cloudformation is now ever discouraged, as it is not very maintainable, but more importantly, it requires knowledge of a syntax that is not exactly easy (enough on yaml or json), so it is recommended to use CDK.
Another consideration is that as much as CDK is tied to AWS, the fact that it is written in a programming language then allows for integration with Pulumi or Terraform (if using cdktf) with the respective language, so that they can work in synergy.
The CDK community is very active, mainly because you will see tutorials or other things based on Typescript, but the constructs are the same for all supported languages.
The following site constructs offers basic constructs/snippets to implement your services.