DEV Community: Harsh Vardhan Singh

Create a cross-account glue Job using AWS CDK

Harsh Vardhan Singh — Tue, 31 Dec 2024 10:11:29 +0000

AWS Glue is a powerful service for data integration and ETL (Extract, Transform, Load) workloads, making it easier to prepare and transform data for analytics. If you’re looking to automate the creation of Glue jobs using Infrastructure as Code (IaC), AWS CDK (Cloud Development Kit) is a great choice. In this post, we’ll walk through the process of defining and deploying an AWS Glue job using AWS CDK. We will be creating a job that can connect to cross-account RDS cluster and execute an etl scripts.

Note : We will not be covering AWS CLI and CDK package setup in this article.

Step 1: Define the VPC stack for Glue Job

export class GlueVpcStack extends DeploymentStack {
  public readonly vpc: Vpc;
  public readonly vpcDefaultSecurityGroupId: string;
  constructor(scope: Construct, id: string, props: DeploymentStackProps) {
    super(scope, id, props);

    const vpc = new Vpc(this, 'VPCForGlue', {
      ipAddresses: IpAddresses.cidr(Vpc.DEFAULT_CIDR_RANGE),
      subnetConfiguration: [
        {
          cidrMask: 24,
          name: 'Public',
          subnetType: SubnetType.PUBLIC,
        },
        {
          cidrMask: 24,
          name: 'Private',
          subnetType: SubnetType.PRIVATE_WITH_EGRESS,
        },
      ],
      natGateways: 1,
    });

    vpc.addGatewayEndpoint('S3GatewayEndpoint', {
      service: GatewayVpcEndpointAwsService.S3,
    });

    vpc.addInterfaceEndpoint('SecretsManagerEndpoint', {
      service: InterfaceVpcEndpointAwsService.SECRETS_MANAGER,
    });

    const vpcDefaultSecurityGroup = SecurityGroup.fromSecurityGroupId(
      this,
      'SecurityGroup',
      vpc.vpcDefaultSecurityGroup,
      {
        allowAllOutbound: false,
        mutable: true,
      },
    );
    this.vpc = vpc;
    this.vpcDefaultSecurityGroupId = vpc.vpcDefaultSecurityGroup;

    vpcDefaultSecurityGroup.addEgressRule(Peer.anyIpv4(), Port.allTraffic());
    vpcDefaultSecurityGroup.addIngressRule(Peer.securityGroupId(this.vpcDefaultSecurityGroupId), Port.allTraffic());
  }
}

Note: We must update the default security group of the VPC to include a self-referencing inbound rule and an outbound rule to allow all traffic from all ports. Later, we attach this security group to an AWS Glue connection to let network interfaces set up by AWS Glue communicate with each other within a private subnet.

Step 2: Define stack for Glue Job and Glue connection

export class InfraStack extends DeploymentStack {
  constructor(scope: Construct, id: string, props: DeploymentStackProps, stageName: string) {
    super(scope, id, props);

    const vpcStack = new GlueVpcStack(this, id + '-VPC', props);

//Creating an IAM role to let AWS Glue access required service 
    const glueRole = new Role(this, id + '-GlueJobsRole', {
      roleName: 'GlueJobsRole-' + stageName,
      assumedBy: new ServicePrincipal('glue.amazonaws.com'),
    });
    glueRole.addManagedPolicy(ManagedPolicy.fromAwsManagedPolicyName('AmazonS3FullAccess'));
    glueRole.addManagedPolicy(ManagedPolicy.fromAwsManagedPolicyName('service-role/AWSGlueServiceRole'));
    glueRole.addManagedPolicy(ManagedPolicy.fromAwsManagedPolicyName('SecretsManagerReadWrite'));

//Creating an AWS Glue connection 
    const glueConnection = new CfnConnection(this, 'GlueConnection', {
      catalogId: this.account,
      connectionInput: {
        connectionType: 'JDBC',
        connectionProperties: {
          JDBC_CONNECTION_URL: "jdbcUrl",
          SECRET_ID: "secretId", //secret that has DB credentials, we need to create this manually if 
          JDBC_ENFORCE_SSL: true,
        },
        physicalConnectionRequirements: {
          securityGroupIdList: [vpcStack.vpcDefaultSecurityGroupId],
          subnetId: vpcStack.vpc.privateSubnets[0].subnetId,
          availabilityZone: vpcStack.vpc.privateSubnets[0].availabilityZone,
        },
        name: 'GlueConnection',
        description: 'GlueConnection',
      },
    });

    //Creating a bucket to keep scripts run in job
    const testBucket = this.createBucket(id.toLowerCase() + '-testgluejobscripts', id + '-testgluejobscripts');
    new BucketDeployment(this, 'DeployTestScripts', {
      sources: [Source.asset('test_glue_job_scripts')], //This folder should be present under root of CDK package
      destinationBucket: testBucket,
    });

    const job = new CfnJob(this, 'TestGlueJob', {
      name: 'TestGlueJob',
      role: glueRole.roleArn,
      command: {
        name: 'pythonshell',
        pythonVersion: '3.9',
        scriptLocation: `s3://<bucket>/script_name.py`,
      },
      glueVersion: '4.0',
      executionProperty: {
        maxConcurrentRuns: 1,
      },
      connections: {
        connections: ['GlueConnection'], //This should match with the  name set inside new CfnConnection() constructor
      },
    });
  }

  private createBucket(name: string, id: string) {
    return new Bucket(this, id, {
      enforceSSL: true,
      bucketName: name,
    });
  }
}

Step 3: Allow Amazon RDS to accept network traffic from AWS Glue

For this, we update the security group attached to the Amazon RDS cluster, and whitelist the Elastic IP address attached to the NAT gateway for the AWS Glue VPC.

Step 4: Deploy the CDK Stack

cdk deploy

Step 5: Verify the Glue Job

Once the deployment is complete, navigate to the AWS Glue Console to verify that the job has been created. The job should appear with the specified configurations and script. You can run the job from console and check expected outcome.

Resources

Running a python script as a standalone task in ECS

Harsh Vardhan Singh — Sun, 09 Jun 2024 08:58:56 +0000

Architecture

Prepare docker Image

Dockerfile - A Dockerfile is a text file that contains a series of instructions and commands used to build a Docker image. It specifies the operating system, application code, dependencies, environment variables, and other necessary configurations to create a Docker image.
Docker Image - A Docker image is a lightweight, standalone, executable package that includes everything needed to run a piece of software, including the code, runtime, libraries, environment variables, and configurations. It serves as a template for creating Docker containers. An image is built from a Dockerfile.
Docker Container - A Docker container is a runtime instance of a Docker image. It is a lightweight, standalone, and executable unit that runs the software defined in the Docker image. Containers are used to run applications in isolation from the host system and other containers.

Create a python script

Create file demp.py in a new directory LightningTalk and add code in it.

Create a docker file

Create a file docker file in same directory



touch Dockerfile

Add following code in Dockerfile



FROM public.ecr.aws/docker/library/python:3

WORKDIR /home/harsh/workplace/LightningTalk

COPY requirements.txt ./
RUN pip install --no-cache-dir -r requirements.txt

COPY . .

CMD [ "python", "./demo.py" ]

We are using python base image from ECR Public Gallery https://gallery.ecr.aws/docker/library/python/?page=1

Build Docker Image



docker build --platform linux/amd64 -t lightning-talk-image:test .

Run container for testing



docker run -it --rm --name lightning-talk-task lightning-talk-image:test

Create ECR Registry and push image to ECR

Run the get-login-password command to authenticate the Docker CLI to your Amazon ECR registry.



aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin $AWSACCOUNTID.dkr.ecr.us-east-1.amazonaws.com

After you have authenticated to an Amazon ECR registry with this command, you can use the client to push and pull images from that registry

Create a repository in Amazon ECR using the create-repository command.



aws ecr create-repository --repository-name python-images --region us-east-1 --image-scanning-configuration scanOnPush=true --image-tag-mutability MUTABLE

Run the docker tag command to tag your local image into your Amazon ECR repository as the latest version. Copy the repositoryUri from the output in the previous step.



docker tag lightning-talk-image:test $AWSACCOUNTID.dkr.ecr.us-east-1.amazonaws.com/python-images:latest

Run the docker push command to deploy your local image to the Amazon ECR repository. Make sure to include :latest at the end of the repository URI.



docker push $AWSACCOUNTID.dkr.ecr.us-east-1.amazonaws.com/python-images:latest

Create ECS Resources and run the task

Create a new cluster



aws ecs create-cluster --cluster-name lightning-talk-cluster

Register a Task Definition

Before you can run a task on your ECS cluster, you must register a task definition. Task definitions are lists of containers grouped together.



aws ecs register-task-definition --cli-input-json file://./fargate-task.json

Before running above command we need to save the task definition JSON as a file fargate-task.json.



{
    "family": "python-tasks",
    "containerDefinitions": [
        {
            "name": "lightning-talk",
            "image": "$AWSACCOUNTID.dkr.ecr.us-east-1.amazonaws.com/python-images:latest",
            "cpu": 256,
            "memory": 2048,
            "portMappings": [
                {
                    "containerPort": 80,
                    "hostPort": 80,
                    "protocol": "tcp"
                }
            ],
            "essential": true,
            "environment": [],
            "environmentFiles": [],
            "mountPoints": [],
            "volumesFrom": [],
            "ulimits": [],
            "logConfiguration": {
                "logDriver": "awslogs",
                "options": {
                    "awslogs-group": "/ecs/python-tasks",
                    "awslogs-create-group": "true",
                    "awslogs-region": "us-east-1",
                    "awslogs-stream-prefix": "ecs"
                },
                "secretOptions": []
            },
            "systemControls": []
        }
    ],
    "executionRoleArn": "arn:aws:iam::$AWSACCOUNTID:role/ecsTaskExecutionRole",
    "networkMode": "awsvpc",
    "requiresCompatibilities": [
        "FARGATE"
    ],
    "cpu": "256",
    "memory": "2048"
}

Running a task

Run following task



aws ecs run-task --cluster lightning-talk-cluster --task-definition python-tasks --launch-type FARGATE --network-configuration 'awsvpcConfiguration={subnets=["subnet-xxxxx","subnet-xxxxx"],securityGroups=["sg-xxxxxx"],assignPublicIp="ENABLED"}'

To get the values of subnet and subnet we can use values from default VPC and default security group.