DEV Community: Yen Trinh

Mount multiple EFS file system to AWS ECS (Fargate)

Yen Trinh — Mon, 19 Sep 2022 12:07:39 +0000

For example, you want to mount EFS file system to ECS Fargate as bellow:

Fargate_1: mount EFS_A, EFS_B
Fargate_2: mount EFS_C, EFS_D

Solution:

You can mount EFS file system one by one according to this post:

https://aws.amazon.com/premiumsupport/knowledge-center/ecs-fargate-mount-efs-containers-tasks/

Note that you will have to repeat step 5 to step 12 to mount the second EFS file system

Demo:

via JSON:

{
    "ipcMode": null,
    "executionRoleArn": "arn:aws:iam::<account-id>:role/ecsTaskExecutionRole",
    "containerDefinitions": [
        {
            "dnsSearchDomains": null,
            "environmentFiles": null,
            "logConfiguration": {
                "logDriver": "awslogs",
                "secretOptions": null,
                "options": {
                    "awslogs-group": "/ecs/task2",
                    "awslogs-region": "ap-northeast-1",
                    "awslogs-stream-prefix": "ecs"
                }
            },
            "entryPoint": null,
            "portMappings": [],
            "command": null,
            "linuxParameters": null,
            "cpu": 0,
            "environment": [],
            "resourceRequirements": null,
            "ulimits": null,
            "dnsServers": null,
            "mountPoints": [
                {
                    "readOnly": null,
                    "containerPath": "/efs1",
                    "sourceVolume": "efs1"
                },
                {
                    "readOnly": null,
                    "containerPath": "/efs2",
                    "sourceVolume": "efs2"
                }
            ],
            "workingDirectory": null,
            "secrets": null,
            "dockerSecurityOptions": null,
            "memory": null,
            "memoryReservation": null,
            "volumesFrom": [],
            "stopTimeout": null,
            "image": "nginx:latest",
            "startTimeout": null,
            "firelensConfiguration": null,
            "dependsOn": null,
            "disableNetworking": null,
            "interactive": null,
            "healthCheck": null,
            "essential": true,
            "links": null,
            "hostname": null,
            "extraHosts": null,
            "pseudoTerminal": null,
            "user": null,
            "readonlyRootFilesystem": null,
            "dockerLabels": null,
            "systemControls": null,
            "privileged": null,
            "name": "nginx"
        }
    ],
    "memory": "512",
    "taskRoleArn": "arn:aws:iam::<account_id>:role/ecsTaskExecutionRole",
    "family": "task2",
    "pidMode": null,
    "requiresCompatibilities": [
        "FARGATE"
    ],
    "networkMode": "awsvpc",
    "runtimePlatform": {
        "operatingSystemFamily": "LINUX",
        "cpuArchitecture": null
    },
    "cpu": "256",
    "inferenceAccelerators": [],
    "proxyConfiguration": null,
    "volumes": [
        {
            "fsxWindowsFileServerVolumeConfiguration": null,
            "efsVolumeConfiguration": {
                "transitEncryptionPort": null,
                "fileSystemId": "fs-06f7aa4c25718fxxx",
                "authorizationConfig": {
                    "iam": "DISABLED",
                    "accessPointId": null
                },
                "transitEncryption": "DISABLED",
                "rootDirectory": "/"
            },
            "name": "efs1",
            "host": null,
            "dockerVolumeConfiguration": null
        },
        {
            "fsxWindowsFileServerVolumeConfiguration": null,
            "efsVolumeConfiguration": {
                "transitEncryptionPort": null,
                "fileSystemId": "fs-0a04ba637cxxx0e6b",
                "authorizationConfig": {
                    "iam": "DISABLED",
                    "accessPointId": null
                },
                "transitEncryption": "DISABLED",
                "rootDirectory": "/"
            },
            "name": "efs2",
            "host": null,
            "dockerVolumeConfiguration": null
        }
    ],
    "tags": []
}

Result:

Glue – Athena custom output fixed number Of Files

Yen Trinh — Mon, 19 Sep 2022 12:04:41 +0000

Situation:

When I only use partition clause, there are so many files in S3 bucket which is <1MB, this affect to the query speed and I want to make those become a bigger file.

Solution:

Solution 1: Use Athena "bucketing" method to custom the number of output file.

You can see this AWS blog for more information:
How can I set the number or size of files when I run a CTAS query in Athena?

However, there is one drawback if you use bucketing: Bucketed table do not support INSERT INTO query. Here comes the solution 2.

Solution 2: Use Glue repartition

The context is the same but now I want to use INSERT INTO query.

You can refer to this AWS blog for the procedure:
Build a Data Lake Foundation with AWS Glue and Amazon S3

Note that in the step numbered "13. View the job", we add the following code into the job:

datasource_df = dropnullfields3.repartition(<number of output file you want here>)

right after the line:

dropnullfields3 = DropNullFields.apply(frame = resolvechoice2, transformation_ctx = "dropnullfields3")

and edit the code:

datasink4 = glueContext.write_dynamic_frame.from_options(frame = dropnullfields3, connection_type = "s3", connection_options = {"path": "<your_s3_path>"}, format = "parquet", transformation_ctx = "datasink4")

to:

datasink4 = glueContext.write_dynamic_frame.from_options(frame = datasource_df, connection_type = "s3", connection_options = {"path": "<your_s3_path>"}, format = "parquet", transformation_ctx = "datasink4")

If you want to know more about Glue repartition:

Try querying with Athena
Create table:

CREATE EXTERNAL TABLE IF NOT EXISTS demo_query (
  dispatching_base_num string,
  pickup_date string,
  locationid bigint)
STORED AS PARQUET
LOCATION 's3://athena-examples/parquet/'
tblproperties ("parquet.compress"="SNAPPY");

Try to insert:

insert into demo_query ("dispatching_base_num", "pickup_date", "locationid") values ('aa23dtgt', '2020-12-03', 1234);

The insert query now should work. Success!

How to query AWS load balancer log if there are terabytes of logs?

Yen Trinh — Wed, 07 Apr 2021 10:32:43 +0000

I want to query AWS load balancer log to automatically and on schedule send report for me.

I am using Amazon Athena and AWS Lambda to trigger Athena. I created data table based on guide here:

https://docs.aws.amazon.com/athena/latest/ug/application-load-balancer-logs.html

However, I encounter following issues:

1) Logs bucket increases in size day by day. And I notice if Athena query need more than 5 minutes to return result, sometimes, it produce "unknown error"

2) Because the maximum timeout for AWS Lambda function is 15 minutes only. Therefore, I can not continue to increase Lambda function timeout to wait for Athena to return result (if in the case that Athena needs >15 minutes to return result, for example)

Can you guys suggest for me some better solution to solve my problem? I am thinking of using ELK stack but I have no experience in working with ELK, can you show me the advantages and disadvantages of ELK compared to the combo: AWS Lambda + AWS Athena? Thank you!