DevSecOps with AWS- Ephemeral Environments – Creating test Environments On-Demand - Part 1

Level 200 - Intermediate ✳️

Nowadays, the velocity of application development process increases rapidly due to the adoption and maturity of disciplines and practices like Platform engineering, DevSecOps, GitOps,Caos Engineering, AIOps, FinOps, Generative AI and others. Seeing AI create apps in seconds, resolve issues and respond to incidents is each time more frequent and tomorrow will be normal as a cloud some years ago. However, builders still have many challenges around the security and governance, the technical debt, focus on business case, learn, and innovate faster and adopt those practices and technologies correctly.

Using the IaC at scale enables practices to apply key pillars of Operational excellence, cost optimization, sustainability, and security. In this series you can learn how apply an Enterprise CI/CD best practice and the design principles according to the Well-Architected Framework,create Test Environments On-demand, using **Azure DevOps, AWS, and CDK **.
Having a single staging environment before production is a common practice, however, is a big disadvantage because it means that developer must either test all their features at once or they must enter a queue, to solve this challenge some companies enable multiple and permanent staging environments for multiple squats for testing in parallel and recurring to some practices as scheduling environments, statics environments adding extra maintenance effort to the team responsible for test environments, increasing the costs due the idle time because those environments are not used all time.

The challenge

Imagine that your company is modernizing the core business application applying practices such DDD and microservices in AWS cloud, some microservices uses synchronous communication and others asynchronous communication, the developer team require test their features for respective services, but these services also depend on others and no all teams deploy features with the same velocity. Currently, the DevSecOps teams enable custom environments for testing with schedule task that recreate the environments every day, every morning for each team. The company needs to minimize the infrastructure costs but also reduce the operational overload and increase security. The main AWS services to support the new solution are ECS, VPC, ALB, RDS.

So, how can you solve this problem? 😐

Read DevSecOps with AWS- IaC at scale - Getting started

First, consider an operation model based on Decentralized DevOps, so the Central Platform team must codify infrastructure standards and best practices with reusable modules or packages.
Second, publish the packages or modules to the private module registry. Finally, the developer team use the packages using interfaces and self-services capabilities such as CLIs, API or UI.

Now, according to the definition of an ephemeral environment:

“A temporary, isolated, and replicable version of a software application that is used for testing, previewing, or collaborating on new features or changes. Ephemeral environments are created on demand and discarded after use, saving time and resources. Ephemeral environments can improve the software development cycle by accelerating the feedback loop, enhancing the quality and security of the software, and reducing the bottlenecks and conflicts in the staging environment.”

Instead of having a predefined number of static and permanent environments, your pipeline workflow each time a Pull Request is created by a developer must provisioning a new test environment with contents of that Pull Request. In other words, use ephemeral environments to get dynamic tests environments. In this way each developer can test in isolation without any conflicts with what other developers are doing, you pay for the resources of test environments only while you use them, since the test environments are discarded at the end there is nothing to maintain or clean up.

Note: you can combine both schedule options and dynamic environments.

The diagram below summarizes the scenario.

Scenarios

There are many scenarios for this use case:
• There is one unique test account for all squats.
• Each squat has a test account.
• Each developer has a test account.
• There is a central account with core resources such as databases and one of the previous scenarios.

Hands On - There is one unique test account for all squats

Overview architecture

The following image depicts the multi-availability zone deployment in an test environment, a DevSecOps CI/CD supported by Azure DevOps, and shared Account for ECR images.

Key points

• The components will be grouped into core environment (VPC, ALB, ECS) and services environment (services definitions, listeners, and rules).
• Each pipeline could deploy one or more services in the same account for reusing the ALB.
• Each group of services use a specific range of ports for
• CDK is the IaC tool, and all stacks and constructs will be written using typescript.
• The CI/CD system just orchestrated the deployment but the dependencies for between IaC layers must keep with the native way for the IaC Tools.
• Consider that your developers would like to test all processes locally.

Requirements

• CDK >= 2.94.0
• AWS CLI >= 2.7.0
• checkov >= 2.1.229
• node >= v18.17.1
• Visual Studio Code
• Docker >= 24.0.6

AWS Services

• Amazon Elastic Container Service (Amazon ECS) is a fully managed container orchestration service that helps you easily deploy, manage, and scale containerized applications.
• AWS Fargate is a serverless, pay-as-you-go compute engine that lets you focus on building applications without managing servers. AWS Fargate is compatible with both Amazon Elastic Container Service (Amazon ECS) and Amazon Elastic Kubernetes Service (Amazon EKS). Select any OCI-compliant container image, define memory and compute resources, and run the container with serverless compute.
• Amazon Virtual Private Cloud (Amazon VPC) gives you full control over your virtual networking environment, including resource placement, connectivity, and security.
• Amazon CloudWatch collects and visualizes real-time logs, metrics, and event data in automated dashboards to streamline your infrastructure and application maintenance.
• Elastic Load Balancing (ELB): automatically distributes incoming application traffic across multiple targets and virtual appliances in one or more Availability Zones (AZs).

The environment setup

The environment setup for integrate Azure DevOps with AWS and CDK deployments is the same configuration present in the blog DevSecOps with AWS- Integrate Azure DevOps for CDK deployments Part-1

You can find the core environment code here:

velez94 / cdkv2_ephemeral_environment_core

Repository for ephemeral environmente core using CDK

Ephemeral environment Core

This is a demo project for CDK development with TypeScript.

The cdk.json file tells the CDK Toolkit how to execute your app.

Code Diagram

High level Architecture

How to

1- Set environment vars or using aws cli profiles for deployment:

export CDK_DEPLOY_ACCOUNT='123456789012'
export CDK_DEPLOY_REGION='us-east-2'

2- Set properties in environment-properties.json

{
  "environment": {
    "name": "cdk-ecs-env-demo",
    "inputs": {
      "env": "dev",
      "vpc_cidr_block": "10.0.0.0/16",
      "nat_provider": "instance",
      "nat_gateways": 1,
      "ec2_capacity": false,
      "ec2_instance_type": "t3.medium",
      "allow_ecs_exec": true,
      "enhanced_cluster_monitoring": true,
      "service_discovery_namespace": "demo.svc",
      "load_balanced": true,
      "load_balanced_public": true
      
    }
  }
}

3- synth and deploy using CDK

Useful commands

• npm run build compile typescript to js
• npm run watch watch for changes and compile
• …

View on GitHub

The first step is to create a stack for core services, suppose that the core will be used every day from Monday to Friday between office hours, the project structure is:

.
├── README.md
├── azure-pipelines.yml
├── bin
├── cdk.context.json
├── cdk.json
├── cdk.out
├── diagram.dot
├── docs
├── environment-properties.json
├── environment.d.ts
├── jest.config.js
├── lib
├── node_modules
├── package-lock.json
├── package.json
├── test
└── tsconfig.json

6 directories, 11 files

The file environment-properties.json defines the properties for VPC, load balancer and ECS Cluster.

{
  "environment": {
    "name": "cdk-ecs-env-demo",
    "inputs": {
      "env": "dev",
      "vpc_cidr_block": "10.0.0.0/16",
      "nat_provider": "instance",
      "nat_gateways": 1,
      "ec2_capacity": false,
      "ec2_instance_type": "t3.medium",
      "allow_ecs_exec": true,
      "enhanced_cluster_monitoring": true,
      "service_discovery_namespace": "demo.svc",
      "load_balanced": true,
      "load_balanced_public": true

    }
  }
}

For other hand, there are one stack and one construct: cdkv2_ephemeral_environment_core-stack.ts and cdkv2_ephemeral_environment_alb_construct.ts.
The outputs are exported via Cloudformation, for example,

...
 new CfnOutput(this, "LBDNSName", {
      value: this.loadBalancer.loadBalancerDnsName,
      exportName: `LBDNSName-${props.stackName}`,
    });

    new CfnOutput(this, "ARNALB", {
      value: this.loadBalancer.loadBalancerArn,
      exportName: `ARNALB-${props.stackName}`,
    });

    new CfnOutput(this, "SGALB", {
      value: this.lbSecGrp.securityGroupId,
      exportName: `SGALB-${props.stackName}`,
    });

...

This creates a lose coupling between layers and enables the resource utilization for other stacks.

The pipeline

velez94 / cdkv2_pipelines_azure_devops

Repository with CDK pipelines definitios in yaml files for azure devops.

CDK Pipelines in azure DevOps

This repository contains de CDK pipelines definitions for integrating azure DevOps and AWS, based on simple authentication and multi account setup in AWS.

Architecture Diagram

Requirements

• Bootstrap accounts for CDK deployments.
• A Service Connection with right permissions.

How to

1. Create a variable group with values for environments, for example:

az pipelines variable-group create --name cdk_pipelines_delivery --variables dev_account=123456789012 dev_region=us-east-2 --authorize true --description "Group for lab Pipelines Delivery" --project Delivery

2. Create an azure pipeline file into cdk project to use these templates, for example:

# File: azure-pipelines-c#.yml
variables: 
- group: 'ephemeral_envrionments_demo'


resources:
  repositories:
    - repository: cdk_pipelines
      type: git
      name: DevSecOps/cdk_pipelines
trigger:
- master

pool:
  vmImage: ubuntu-latest


stages:
- template: templates/ci_cd.yaml@cdk_pipelines  # Template reference

  parameters:
    Project: ephemeral_env_core
    Language: typescript
    Action: deploy
    ServiceConnection: EphemeralEnvDeployment

Learn how in:

• Azure…

View on GitHub

Please visit DevSecOps with AWS- Integrate Azure DevOps for CDK deployments Part-1 and DevSecOps with AWS- Integrate Azure DevOps for CDK deployments Part-2

For this example, Azure DevOps is the CI/CD tool and based on previous post the simple configuration is:

# File: azure-pipelines.yml
variables: 
- group: 'ephemeral_envrionments_demo'


resources:
  repositories:
    - repository: cdk_pipelines
      type: git
      name: DevSecOps/cdk_pipelines
trigger:
- master

pool:
  vmImage: ubuntu-latest


stages:
- template: templates/ci_cd.yaml@cdk_pipelines  # Template reference

  parameters:
    Project: ephemeral_env_core
    Language: typescript
    Action: deploy
    ServiceConnection: EphemeralEnvDeployment

The following image depicts the pipeline execution for deploying core resources:

Thanks for reading and sharing! 😊

Comments

[Piyali Debroy]

Helpful information.