DEV Community: serverless.ninja

How to build a self-service for your tenant AWS accounts?

Jakub Gaj — Thu, 19 Jan 2023 09:34:19 +0000

Introduction

Quite recently I got a great opportunity to speak at a community-led conference AWS Community Day as part of AWS Pop-up Hub Warsaw. In my brief presentation (in Polish) I've tried to show how to build a self-service for tenant AWS accounts, based on AWS Service Catalog and AWS Organizations. This article is a follow-up to my presentation on how to automate deployments of predefined AWS resources across multiple AWS accounts in your organization.

Service Catalog

AWS Service Catalog is a perfect service to build either a simple or more complex self-service for your AWS tenants. For example, you can prepare a well-designed 3-tier VPC, package it as a CF template and present it as an SC product, so your end users can simply kick off the VPC deployments in their AWS accounts by simply launching the SC product and providing some required parameters (i.e. CIDR block).

Portfolios

SC portfolios allow you to organize, manage, and distribute cloud resources for your end users. You can share the portfolios with other AWS accounts and allow the admins of those accounts to distribute the portfolios with additional restrictions.

Products

SC products are version-friendly infrastructure-as-code templates added to portfolios for end users to provision and create AWS resources. A product can be either a single EC2 instance, S3 bucket, EKS cluster, full-blown VPC networking, multi-tier web application, or anything in between. It's simply a predefined CloudFormation template or a template imported from an existing CF stack.

Some sample CF templates for SC products:
https://github.com/chmoora/aws-self-service/blob/master/src/cf-product-s3.yaml
https://github.com/chmoora/aws-self-service/blob/master/src/cf-product-ssm.yaml

Constraints

SC constraints are certain limits that you can place on product-portfolio associations that allow you to manage minimal launch permissions or other optional actions that end users can perform on products. The most useful I usually find are:

launch constraints - for specifying a dedicated IAM role in the target account, which will be assumed by SC service to deploy CF stacks
template constraints - for limiting the values of CF parameters available to your end users, for example, a list of EC2 instance types
tag update constraints - used to allow/disallow end users to update tags on resources provisioned by the SC products

Sharing

Your SC portfolios can be shared with users/roles inside a single AWS account, as well as with selected AWS accounts or a whole AWS organization. This also includes any external AWS accounts or other AWS organizations, so you can technically build a single self-service for multiple customers.

The following diagram shows a sample structure of related AWS accounts in a single organization: Management account (Organizations), Provisioning account (delegated Admin account for Service Catalog) and some Project accounts (spoke/tenant accounts). Additional IAM role(s) used as launch constraints need to be defined in the spoke accounts, which you can simply deploy with CloudFormation StackSets.

CDK Constructs

To simplify the deployment of the Service Catalog resources in your Management or Provisioning account you can define them in a CDK application.

https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_servicecatalog-readme.html

CloudFormation Product

Your products can be defined in CDK app in two ways. The first one is with the CloudFormationProduct construct: a CloudFormation template asset, stored either locally or remotely (S3, CodeCommit, GitHub). You have the template and you're providing its location. Easy, let's move on.

Product Stack

The second way is the ProductStack construct: a special CDK stack, where you define the resources you want to define for your SC product. This CDK stack will not be deployed as a CF stack, but rather only synthesized to a local JSON asset and consumed as a source template of the SC product. A pretty cool feature of CDK itself.

export class ScProductBucket extends ProductStack {

  constructor(scope: Construct, id: string) {
    super(scope, id);

    new s3.Bucket(this, 'PrivateBucket', {
      blockPublicAccess: s3.BlockPublicAccess.BLOCK_ALL,
      encryption: s3.BucketEncryption.KMS_MANAGED,
      enforceSSL: true,
      versioned: true
    });
  }
}

The full code for Product Stack:
https://github.com/chmoora/aws-self-service/blob/master/lib/sc-products-stack.ts

Portfolio

Defining the Portfolio is pretty straightforward as well. The same applies to adding products and their versions to the portfolio. The sharing part is a bit more tricky.

AWS CDK and CloudFormation only support sharing with individual AWS accounts (by account ID) with the method shareWithAccount. Unfortunately, sharing with Organization or Organizational Unit is currently not supported (as of CDK release v2.60.0). However, there's always a workaround, more or less hacky.

This is a perfect opportunity to show you another interesting CDK construct, namely the AwsCustomResource. It's a singleton Lambda function, which can be used to extend the functionality of the CDK and/or CloudFormation. You can provide your code of course, but if you just need to get a response from some AWS API, the function code will be generated for you.

In my use case, I want to share the SC portfolio with my whole organization. To do this I'd need my organization's ID to create the portfolio share. I'm using here two AWS API calls to Organizations and Service Catalog services, respectively:

DescribeOrganization - to get dynamically the ID of my AWS Organization
CreatePortfolioShare - to share the portfolio with the whole Organization

Alternatively, I could provide my organization's ID statically, for example by using the CDK context file cdk.context.json. This would probably make more sense if I'd need to share the portfolio(s) with multiple external organizations and would need to have a list of IDs, then loop through them when creating the shares.

The code snippet below defines Custom Resource calling out AWS API call:

const describeOrg = new cr.AwsCustomResource(this, 'DescOrg', {
  onUpdate: {
    service: 'Organizations',
    action: 'describeOrganization',
    region: 'us-east-1',
    physicalResourceId:cr.PhysicalResourceId.of(
      Date.now().toString()
    ),
  },
  policy: cr.AwsCustomResourcePolicy.fromSdkCalls({
    resources: cr.AwsCustomResourcePolicy.ANY_RESOURCE,
  }),
});

const orgId = describeOrg.getResponseField('Organization.Id');

The next important step is defining the products and their versions inside the portfolio and setting IAM role launch constraint, which can be simply done in the CDK with addProduct() and setLocalLaunchRoleName() methods, respectively. In my CDK stack, I'm building a list of products' CF templates with fromProductStack() method, which will be imported from the synthesized template produced by the ProductStack.

The full code for all SC-related resources is available here:
https://github.com/chmoora/aws-self-service/blob/master/lib/sc-self-service-stack.ts

ITSM Integrations

Another interesting aspect of the AWS Service Catalog is integration with IT Service Management (ITSM) systems, which can be done with AWS Service Management Connectors (SMC). The connectors enable you to provision, manage, and operate native AWS resources directly from your ITSM system. Currently, supported integrations include ServiceNow Service Catalog and Atlassian Jira Service Management.

AWS Service Management Connector for ServiceNow:
https://store.servicenow.com/sn_appstore_store.do#!/store/application/f0b117a3db32320093a7d7a0cf961912/

AWS Service Management Connector for JSM:
https://marketplace.atlassian.com/apps/1221283/aws-service-management-connector-for-jsm-cloud?tab=overview&hosting=cloud

Summary

I hope this article will inspire you to build a self-service solution in your organization. It would be great if you'd share in the comments how have you organised your portfolios and what useful products have you designed for your end users in tenant AWS accounts. Have fun!

Resources

Self-Service (CDK Application)
https://github.com/chmoora/aws-self-service

IAM Roles (CDK Application)
https://github.com/chmoora/aws-iam-roles

AWS Service Catalog Construct Library (CDK)
https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_servicecatalog-readme.html

AWS Service Catalog Administrator Guide
https://docs.aws.amazon.com/servicecatalog/latest/adminguide/

AWS Service Catalog User Guide
https://docs.aws.amazon.com/servicecatalog/latest/userguide/end-user-console.html

AWS Service Management Connectors
https://docs.aws.amazon.com/smc/latest/ag/overview.html

How to prototype cloud solutions with AWS CDK?

Jakub Gaj — Tue, 03 Jan 2023 12:06:24 +0000

Prerequisites

This article assumes that you are somehow familiar with the AWS Cloud Development Kit framework and its concepts. Otherwise, I'd highly recommend you to check the following resources before continuing:

AWS CDK v2 Developer Guide
https://docs.aws.amazon.com/cdk/v2/guide
AWS CDK Workshop
https://cdkworkshop.com/

Infrastructure as Code

As a cloud architect, I have to design different cloud-based solutions almost on a daily basis. Depending on the requirements, the size & complexity of each solution might differ, of course. What I usually need is a proof of concept, a proof that the overall idea will eventually work as designed. Also, if selected AWS services can interact with each other out-of-the-box, or will I have to glue some of the bits together in low-code or some-code fashion, for example with custom Lambda functions.

Clicking things out straight on the AWS Management Console is probably the most common way of prototyping in a sandbox environment. I'm sure we've all been there. AWS just makes it super easy to connect different services, in most cases at least.

One of the pros of this approach is that the AWS console wizards can automatically create any necessary IAM roles and related permissions embedded in IAM policies. Downside is that at some point I might need to write it all down as Infrastructure as Code, for obvious reasons. CloudFormation is a native choice for any AWS developer, but building more complex solutions requires very fast fingers to generate that beautiful YAML code (or the ugly JSON, if you're hardcore enough).

I still have a repository of own CloudFormation templates and snippets, which I can reuse anytime in a modular way in combination with CF features like macros, nested stacks, stack sets, etc. The problem is that any larger solution generates tons of YAML code, which is quite hard to maintain and refactor.

By entering the world of AWS CDK, I could easily avoid some of the CF templates horror stories. In the beginning I didn't like the framework at all, mostly because it was a bit too overwhelming to work with. I've decided to give it another try when Python became one of the supported languages. I must admit, it took me few tries to get used to CDK's concept and to discovered the true potential of this framework.

Today I can say honestly that I do enjoy developing my AWS solutions in CDK. They usually end up as multiple CDK apps with multiple CDK stacks and deployed to multiple AWS accounts and/or regions. My CDK apps get deployed automatically from different CI/CD pipelines, as CDK can handle deployments of CloudFormation stacks pretty well in very automated fashion.

Constructs and patterns

For me the true power of the AWS CDK framework lays in so called CDK Constructs. My current solutions portfolio consists of CDK constructs for commonly used architectural patterns, which I often use as a skeleton for any AWS-based infrastructures. A perfect example would be 3-tier VPC networking with all the bells and whistles, conditionally deployed or not depending on the target environment (dev, prod, etc). Sounds familiar?

In this post I wanted to show you some of the publicly available repositories of architectural patterns and solution constructs, which I find very useful, especially for some quick & dirty developments. The constructs basically allow for very rapid prototyping, as most of them deploy all the necessary infrastructure resources, or allow me to specify existing ones instead (for example, the VPC).

AWS Construct Library

This is basically a list of L3 Constructs, also just called patterns. They are part of AWS CDK API Reference and can be found under aws-cdk-lib.aws_{service}_patterns.

https://docs.aws.amazon.com/cdk/api/v2/docs/aws-construct-library.html

Application Load Balanced Fargate Service

This is one of my favourite construct, as it deploys quite common pattern: Application Load Balancer with ECS service running in ECS Fargate cluster.

https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_ecs_patterns.ApplicationLoadBalancedFargateService.html

const domainName = 'aws.chmoora.io';

const hostedZone: r53.IHostedZone = r53.HostedZone.fromLookup(this, 'Zone', {
  domainName: domainName,
  privateZone: false
});

const vpc: ec2.IVpc = ec2.vpc.fromLookup(this, 'Vpc', {
  vpcName: 'myapp-vpc',
  isDefault: false,
});

new ecs.ApplicationLoadBalancedFargateService(this, 'Fargate', {
  vpc: vpc,
  desiredCount: 1,
  memoryLimitMiB: 1024,
  cpu: 512,
  assignPublicIp: false,
  protocol: elbv2.ApplicationProtocol.HTTPS,
  domainName: `myapp.${domainName}`,
  domainZone: hostedZone,
  taskImageOptions: {
    image: ecs.ContainerImage.fromRegistry("nginx:latest"),
  },
  taskSubnets: {
    subnetType: ec2.SubnetType.PRIVATE_WITH_EGRESS
  },
});

With default values CDK will generate CF resources for a new VPC, ECS cluster, ECS service, Application Load Balancer, etc. I can enforce HTTPS protocol to generate a TLS certificate stored in Certificate Manager, based on custom domain name defined in Route 53 public Hosted Zone. And that's it! Here's my containerised Nginx server running on ECS cluster behind ALB with HTTPS listener.

In the above example I'm relying on existing VPC and Route 53 Hosted Zone, which I'm importing inside my CDK stack. How does it work exactly? Well, CDK will lookup in my target AWS account(s) for specific AWS resources, then will cache their ARNs in cdk.context.json file, which is part of the CDK app. Pretty cool, huh? :)

Also, I can make my solution a bit more production ready, for example by adding some CloudFront distribution and WAF WebACLs for extra layer of protection.

AWS Solutions Constructs

This AWS CDK extension provides some multi-service, well-architected patterns for quickly defining whole solutions in code. It's nothing more but a browsable and searchable library of open-source patterns, which you can use in your CDK projects.

https://aws.amazon.com/solutions/constructs/patterns/
https://docs.aws.amazon.com/solutions/latest/constructs

The library contains some of the commonly used patterns, for example:

API Gateway to Lambda
https://docs.aws.amazon.com/solutions/latest/constructs/aws-apigateway-lambda.html

EventBridge to Lambda
https://docs.aws.amazon.com/solutions/latest/constructs/aws-eventbridge-lambda.html

EventBridge to Step Functions
https://docs.aws.amazon.com/solutions/latest/constructs/aws-eventbridge-stepfunctions.html

CloudFront to S3
https://docs.aws.amazon.com/solutions/latest/constructs/aws-cloudfront-s3.html

Constructs Hub

Another great source of CDK constructs is the Constructs Hub. This extensive library contains CDK patterns from different publishers, for different programming languages and also patterns for CDK extensions, like CDKCF (CDK for Terraform) or CDK8s (CDK for Kubernetes).

https://constructs.dev/

Temporary Stack

Have you ever forgot to destroy your playground CF stacks on friday afternoon and the AWS resources kept running over weekend, incuring some costs connected to your credit card? We've all been there, trust me :) Wouldn't be nice to be able to automatically destroy your AWS resources deployed with CF stack(s) after specific period of time, let's say after X number of hours or days?

This community provided CDK construct called TempStack by CloudComponents does exactly that. Simply add TimeToLive resource to your CDK stack, which will deploy couple of resources handling the lifecycle of the CF stack, namely Lambda function and EventBridge rule to trigger it on schedule. The absolute coolness!

https://constructs.dev/packages/@cloudcomponents/cdk-temp-stack/

import { Construct } from 'constructs';
import { Stack, StackProps, Duration } from 'aws-cdk-lib';
import { TimeToLive } from '@cloudcomponents/cdk-temp-stack';
import { Vpc, IpAddresses, IpAddresses } from 'aws-cdk-lib/aws-ec2';

export class MyPlaygroundVpcStack extends Stack {
  constructor(scope: Construct, id: string, props?: StackProps) {
    super(scope, id, props);

    new TimeToLive(this, 'Ttl', {
      ttl: Duration.days(1),
    });

    new Vpc(this, 'Vpc', {
      vpcName: 'my-playground-vpc',
      ipAddresses: IpAddresses.cidr('10.10.100.0/24'),
      maxAzs: 2,
      natGateways: 1,
      enableDnsHostnames: true,
      enableDnsSupport: true,
      subnetConfiguration: [
        {
          cidrMask: 26,
          name: 'entry',
          subnetType: SubnetType.PUBLIC,
        },
        {
          cidrMask: 26,
          name: 'app',
          subnetType: SubnetType.PRIVATE_WITH_EGRESS,
        }
      ]
    });

  }
}

Summary

I hope this post will inspire you a bit to play around with CDK constructs in your current or future projects. It would be great if you'd share in the comments any constructs from provided resources, which you've found particularly useful or worth checking out. Also, feel free to share any repositories with your own CDK constructs you've developed & published in the past, I'd love to have a look!

Have fun! :)