Sophisticated use-case of AWS CDK that leverages AWS SDK

by Udaybhaskar Sarma Seetamraju (ToSarma gmail)
Oct 16 2023

One sentence summary

A real-world sophisticated use-case exists for leveraging AWS SDK within AWS CDK code, when deploying cloud-native Non-Kubernetes solutions, within Enterprise-environments where IP-address space is a major constraint.

One Paragraph Summary

For those of us, who do Not work for the leading-edge tech-companies (which is the overwhelming majority of the Fortune 1000), private IP addresses are a fact. That is, we face serious constraints in how many VPCs and AWS Accounts can exist. Also, frequently due to lack of an IPAM system, the ability to automatically create and dispose AWS Accounts is missing. In such Enterprise contexts, as shown in this article, using AWS SDK (within AWS CDK code) leads to far simpler and far more robust solutions, specifically when deploying cloud-native Non-Kubernetes solutions.

Focus Areas & Keywords

DevOps, Enterprise Environments, “Dirty” VPCs, AWS CDK, AWS SDK, CLI Profiles, “Everything is Disposable”.

The main article

Enterprise Context: A unwavering focus on Production

My Philosophical “True North”: Regarding AWS Resources, my perspective is: AWS SDK is for obtaining details on what already exists, whereas AWS CDK (incl. Cloudformation) is about what we need to create. In other words, I personally do Not use AWS SDK to create new AWS Resources, nor do I “mildly” manipulate any pre-existing AWS Resources. FWIW: CDK itself relies on SDK.

I belong to the group of people who do Not like Cloudformation in which each resource has a “Condition:”. The best practice (of maintainability / supportability) is to have Cloudformation-templates (CFT) that simply & clearly tell you what was deployed; It’s easy to predict/understand what will happen if you re-deploy. In many industries, Cloudformation-templates, Stack-drifts and Change-Sets are like god’s gifts, but that is out of scope for this article.

“Corporate Life”: Commonly enough within Enterprise environments, either there is a limited ability, or there is an outright ban on the ability to create “fresh new” VPCs and/or AWS Accounts. An even more important nuance is, this limitation/absence can be quite severe within Production environment/VPCs, even as developers may experience “freedom” within sandbox/PoC/Development/Integration environments.

Per the best practice for designing solutions, we must have all identical environments. So, developers should “copy/simulate” the constraints of Production into all the Non-prod environments.

Use-Case / Scenario: This article is specific to one specific use-case, where CDK-based deployment is done manually (via “cdk” CLI). Personally, I experience this use-case early in every project, when the deployment-architecture diagram (onto AWS) is constantly in flux. The iterative approach using the CI/CD pipeline to improve your CDK is too horribly slow, un-acceptably inefficient and error-prone.

For any of my projects, for 3 specific environments (namely sandbox, PoC and development environments), I rely on manual CDK-CLI based deployments.

The core security needs

• CDK offers a great advantage within DevSecOps pipelines, something we should all take FULL advantage of. Via CDK-code, we can very easily ensure all IAM Roles and Network-security components are also covered robustly, for each component of overall solution.
• The example in this article focuses solely on AWS Security Groups (SGs), since use-cases to share at least one SG (across various components of a solution) is a common enough. More details on this is throughout this article.
Note: Never share IAM Roles across components of a solution.
• In the early release of a medium/large application, it is very common to have an “Application-wide” Security Group (“AppWide-SG”). This is a good best-practice to remember, as it forces developers to start day-one, in strictly using SGs to limit access to every single component.

What is a Dirty VPC?

• The concept of a “Dirty VPC” is simply an implication of the Enterprise constraint that “One is restricted in creating new VPCs within DevSecOps pipelines”
• The dictionary word “Dirty” has a negative meaning. But, for this article, we will interpret that word to mean that the VPC already has one of more desired AWS resources; in this example, Security-Group(s).
• I love the philosophical approach off "Everything is Disposable". I practice it a lot, including the use of cloud IDEs.
But, “Dirty VPCs” are the proverbial “immovable object” within Enterprise environments.

Some foundational facts

• FYI: Per AWS Official documentation, AWS SDK authenticates (a.k.a. AuthT) based on one of the following 3 mechanisms:
  1. AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_SESSION_TOKEN environment variables.
  2. Based on “--profile” CLI arguments.
  3. Based on “default” AWS Profile in ~/.credentials file.
• FYI: Similarly, AWS CDK relies on the above 3 for AuthT a.k.a. authen*T*ication.
• FYI: Similarly, re: the “Region”, both AWS CDK & AWS SDK determine it either based on environment variables, or “--region” CLI-arg or on the region specified within “default” AWS Profile in ~/.credentials file.
• Note: In an Enterprise work-environment, due to pervasive use of SSO (Single-SignOn), you are highly likely to be using the “--profile” cli-arg.

DRY (Don’t Repeat Yourself) Problem

• Best Practice: For sandbox/PoC/development environments, exclusively for manual deployments, CDK best practices suggest that you use “--profile” CLI-arg (as well as the “—region” CLI-arg.
In contrast, for all other environments, especially within DevSecOps pipelines, you should be exclusively rely on AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_SESSION_TOKEN environment variables. CDK’s & SDK’s best-practices suggests the use of “--profile” CLI-arg and the “--Region” CLI-arg (instead of using “default” AWS Profile).
• Limitation: Per the AWS Support-case 13701110271, dated Sept 2023,
AWS confirmed that CDK uses AWS SDK underneath, but, unfortunately, we can Not get “access” to the SDK.
That implies that our code must explicitly initialize AWS SDK, by providing it the credentials.
• The Issue: the CDK will Not share details on how it successfully-authenticated -- whether via Env-Vars, or via the “--profile” CLI-arg, or via the “default” AWS Profile.
• The PROBLEM: We have a “DRY problem”, when we use “--profile” CLI-arg and the “--Region” CLI-arg (instead of “default” AWS Profile); AWS CDK gets that information, but Not the AWS SDK. 
So, we need to “copy/clone” that information and send it to AWS SDK too (for it to AuthT too).
See code-snippet #2 below, for the implementation.

The official Workaround: AWS-SDK AuthT

Within my CDK code, I mandate 2 additional CLI-arguments as: “--context AWSProfile=...” and 
“--context AWSRegion=...” To enforce these additional cli-args, use code-snippet #1 (see below).
FYI: The above makes your command for manual CDK-deployment look like:-

cdk deploy|synth \
     --profile ${AWSPROFILE} \
     --region ${AWSREGION} \
     --context AWSProfile=${AWSPROFILE} \
     --context AWSRegion=${AWSREGION}

Per the AWS Support-case 13701110271, dated Sept 2023, AWS
FYI: I got 2023-specific confirmation on what I’m recommending within this article: “What you are doing with the context file is the best approach to this.” Please do Not assume this remains true for 2025 & beyond.

Code Snippet #1 - “cdk” CLI’s mandatory CLI args

Within your CDK project, please copy these lines within the only file: “./bin/*.ts”, ideally preceding the lines starting with “new Stack(app, .. {”.

const myAWSProfile = scope.node.tryGetContext('AWSProfile');

if ( ! AWSProfile ) {
    console.error(`!! FATAL-ERROR: CLI-args are missing:-->      --context AWSProfile=....\n\nExiting ...`);
    process.exit(1);
}

const myAWSRegion = app.node.tryGetContext('AWSRegion');

if ( ! AWSRegion ) {
    console.error(`!! FATAL-ERROR: CLI-args are missing:-->      --context AWSRegion =....\n\nExiting ...`);
    process.exit(1);
}

// OPTIONAL CLI-arg
const isDirtyVPC_cliarg :string | undefined = scope.node.tryGetContext('dirtyVPC');
const isDirtyVPC :boolean = (isDirtyVPC_cliarg != undefined) && (isDirtyVPC_cliarg !== "false");
if ( isDirtyVPC ) {
    console.log(`Ok! Deploying to a "dirty" VPC that may already have certain SecurityGroups!`);
}

Code Snippet #2 - AWS-SDK: Authenticating into AWS-API

import * as EC2 from ‘aws-cdk-lib/aws-ec2';
..
..
  const aws_sdk_credentials = {
      credentials: fromIni({
          profile: myAWSProfile,
      }),
      // region: process.env.CDK_DEPLOY_REGION || process.env.CDK_DEFAULT_REGION,
      region: myAWSRegion,
  };

  const stsclient = new STSClient(aws_sdk_credentials);
  const stscommand = new GetCallerIdentityCommand({});
  const stsresponse = stsclient.send(stscommand).then((data) => {
     const AWSProfile_derived = data.Arn?.split('/')[1] || “UnknownProfile1";
     console.log(`Sanity-Check: AWSProfile in ARN = ${AWSProfile_derived}`);
  }

How to keep your CFT simple & highly maintainable

This article wouldn’t be complete without ready-to-use CDK code-snippets, to conditionally/optionally generate resources within your Cloudformation-Template (CFT), after AWS-SDK calls are used to determine those conditions.

In this example (code snippet #3), we use AWS-SDK calls to determine if an Security-Group exists; If missing, the creation of the SG (AWS Security Group) is included in the cloudformation generated by SDK.

Code Snippet #3 - Conditional Cloudformation

import * as EC2 from ‘aws-cdk-lib/aws-ec2';
import { EC2Client } from ‘@aws-sdk/client-ec2';
..
..
async function lookupSecurityGroup( securityGroupName: string, 
                                    aws_sdk_credentials: any, )
              : Promise<string | undefined>
{
    const ec2client = new EC2Client( aws_sdk_credentials );
  ..
  .. // see full version of THIS function -> at bottom of this article.
  ..
    const sgresponse = await ec2client.send( sgcommand );
    if ( !sgresponse.SecurityGroups || sgresponse.SecurityGroups!.length < 1) {
        throw .. ..
    } else {
      return new Promise((res) => res( sgresponse.SecurityGroups[0].GroupId );
    }
} // end of async function.

// now let’s utilize the above function !!

    await lookupSecurityGroup( SGNAME, aws_sdk_credentials )
    .then((sgid) => {
        myAppWideSecurityGroup = EC2.SecurityGroup.fromLookupByName( this,  SGNAME, SGNAME, defaultVpc );
        if ( ! isDirtyVPC ) {
           throw Error( `!! SAFETY-CHECK ERROR !! SG ${SGNAME} already exists.  You did NOT provide CLI-arg "--dirtyVPC".\nExiting with error-code ...` );
        }
     }).catch( (e) => {
        console.log( `Ok. SG Does Not exist. So, creating new SG named ${SGNAME} ...` );
        const mySGProps: EC2.SecurityGroupProps = {
                    vpc: defaultVpc,
                    securityGroupName: SGNAME,
                    description: SGDESC,
                    allowAllOutbound: true,
                    disableInlineRules: true,
        };
        myAppWideSG = new EC2.SecurityGroup( this, SGNAME, mySGProps );
        // commented out: myAppWideSG.addEgressRule( EC2.Peer.anyIpv4(), EC2.Port.allTcp(), "Any Port OUTbound" );

        const cfn_sg = myAppWideSG.node.defaultChild as EC2.CfnSecurityGroup;
        cfn_sg.overrideLogicalId( SGNAME.replace(/-/g, '') ); 

The last line of code removes all hyphens from SGNAME, so the resulting string can be used as Resource-Name within Cloudformation-template.

Note: I'm ignoring addEgressRule since 'allowAllOutbound' is set to true by default; To add customized rules, set allowAllOutbound=false.

Key Takeways

Even though this article focused on a very valid common real-world scenario w.r.t. AWS Security Groups, this article offers a simple clean way to combine AWS-SDK with AWS-CDK, to can design/write CDK code that cleanly checks for a pre-existing component (say, a Fargate ECS-Cluster), and skip the generation of Cloudformation accordingly.

APPENDIX: full-code

import { EC2Client, DescribeVpcsCommand, DescribeSecurityGroupsCommand } from '@aws-sdk/client-ec2';
import { SecurityGroup } from '@aws-sdk/client-ec2';

export async function lookupSecurityGroup( securityGroupName: string,  aws_sdk_credentials: any, ): Promise<string>
{
  const ec2client = new EC2Client( aws_sdk_credentials );
  const filter = {
      Filters: [ { Name: "isDefault",  Values: ["true"] } ],
      MaxResults: 5, // Number("int");    Attention! Minimum value is 5. Maximum value of 1000.
      DryRun: false,
      // VpcIds: [ "vpc-0123456", ],
  };
  const vpccommand = new DescribeVpcsCommand( filter );
  const vpcresponse = await ec2client.send(vpccommand);
  if ( ! vpcresponse.Vpcs || vpcresponse.Vpcs.length < 1 ) {
      throw new Error('No default VPC found');
  }
  const defaultVpcId = vpcresponse.Vpcs![0].VpcId!;
  // DescribeVpcsResult: https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/clients/client-ec2/interfaces/vpc-8.html
  // {  Vpcs: [{
  //       CidrBlock: "STRING_VALUE",
  //       DhcpOptionsId: "STRING_VALUE",
  //       VpcId: "STRING_VALUE",
  //       IsDefault: true || false,
  //       Tags: [ // TagList
  //       .. ..

  // ---------------------------------- 
  const sgfilter = { // DescribeSecurityGroupsRequest
      Filters: [
          { Name: 'vpc-id', Values: [defaultVpcId], },
          { Name: 'group-name', Values: [securityGroupName], },
               // NOTE: Simpler to use "GroupName" instead !!!
      ],
      DryRun: false,
      MaxResults: 5, // Number("int");    Attention!  Minimum value is 5. Maximum value of 1000.
      // GroupIds: [ "sg-12345678", ],
      // GroupNames: [  "STRING_VALUE", ],
      // NextToken: "STRING_VALUE",
  };
  const sgcommand = new DescribeSecurityGroupsCommand( sgfilter );
  const sgresponse = await ec2client.send(sgcommand);
  // { // DescribeSecurityGroupsResult
  //   SecurityGroups: [ // SecurityGroupList
  //     { // SecurityGroup
  //       Description: "STRING_VALUE",
  //       GroupName: "STRING_VALUE",
  if ( ! sgresponse.SecurityGroups || sgresponse.SecurityGroups!.length < 1 ) {
      throw new Error(`No SG found with name ${securityGroupName}`);
  }

  return sgresponse.SecurityGroups![0].GroupId!;
}