DEV Community: Petra Barus

Tracking PHPUnit Results in AWS CodeBuild (with AWS CDK)

Petra Barus — Thu, 23 Apr 2020 12:06:38 +0000

When we implement a testing stage for our code in the CI/CD pipeline, we will need to see the results of the tests and how they are progressing on every commits. In this tutorial I am going to share about how to use Test Reports feature on AWS CodeBuild to display test results from PHPUnit. I am also going to share the AWS CDK code so you can easily replicate it to your other projects/services.

Test Reports is a feature in AWS CodeBuild launched end of last year. It is still in preview at the time I wrote this article, so it might still subject to change. UPDATE: Test Reporting is now generally available.

Before Test Reports, the easiest way to see the test results aside from reading the terminal output is probably by generating the results as HTML and upload it to S3 bucket for you to download. But it will be hard to track the trends from commits to commits. Here I am going to show how the results are visualized by Test Reports.

Requirements

To run this, you will need an AWS Account. If you have a new AWS account, the resources we are going to create here are still under the free tier. You are also going to need Docker, Docker Compose, and AWS CDK installed.

The Code

You can access the sample code I wrote on GitHub. You can clone it or fork it on your GitHub.

Running The Code Locally

To run unit test locally, you can execute

docker build -t phpunit-codebuild-test-reports .
docker run -v ${PWD}:/app phpunit-codebuild-test-reports \
    ./vendor/bin/phpunit

You will see output that the unit test succeed.

PHPUnit 9.1.1 by Sebastian Bergmann and contributors.

..                                                                  2 / 2 (100%)

Time: 342 ms, Memory: 4.00 MB

OK (2 tests, 4 assertions)

Tests Code

The two source code below shows the PHP class that we want to test and the test class itself. Class MyClass has two simple method to add two integers and to concatenate two strings.

<?php

class MyClass {

    public function method1(int $a, int $b): int {
        return $a + $b;
    }

    public function method2(string $a, string $b): string {
        return $a . $b;
    }
}

The test class MyClassTest will have two testcases, each for the method.

<?php

use PHPUnit\Framework\TestCase;

class MyClassTest extends TestCase {

    public function testMethod1() {
        $object = new MyClass();
        $this->assertEquals(2, $object->method1(1, 1), 'Test Method 1 Case 1');
        $this->assertEquals(3, $object->method1(1, 2), 'Test Method 1 Case 2');
    }

    public function testMethod2() {
        $object = new MyClass();
        $this->assertEquals('11', $object->method2('1', '1'), 'Test Method 2 Case 1');
        $this->assertEquals('12', $object->method2('1', '2'), 'Test Method 2 Case 2');
    }
}

In the PHPUnit configuration phpunit.xml.dist, we need to configure the type of the result to JUnit junit so it can be recognized by CodeBuild Test Reports. We can set the target file anywhere as long as it's consistent with what we will put in the CodeBuild configuration. I usually put it inside build directory.

<?xml version="1.0" encoding="UTF-8"?>

<phpunit bootstrap = "vendor/autoload.php">
    <testsuites>
        <testsuite name="Project Test Suite">
            <directory>tests</directory>
        </testsuite>
    </testsuites>
    <filter>
        <whitelist>
            <directory suffix=".php">src/</directory>
        </whitelist>
    </filter>
    <logging>
        <log type="junit" target="build/reports.xml"/>
    </logging>
</phpunit>

CodeBuild Configuration

CodeBuild uses Build Specification or buildspec to configure what commands to run and other settings like runtime. The buildspec.yml configuration for CodeBuild will look like the code below. We configure the location of the test report in the reports directive. We set the test report to use the generated reports file that we already configured before build/reports.xml. This test report will use JUnit XML report format.

version: '0.2'
phases:
  install:
    runtime-versions:
      php: '7.3'
    commands:
    - composer install
  build:
    commands:
    - ./vendor/bin/phpunit
reports:
  test-report:
    files:
      - build/reports.xml
    file-format: JunitXml

AWS CDK Code

By using AWS CDK we can model and track our AWS infrastructure using familiar programming language like Typescript, Javascript, Java, .NET, and Python. Because most PHP projects use Javascript, we are going to use Typescript for our AWS CDK code. It doesn't need a lot of requirements to be able to use Typescript if you already have Node installed for your PHP projects. The full CDK code can be read here since I am going to explain part per part.

In this tutorial we are not going to build whole CI/CD Pipeline using AWS CodePipeline. We are just going to use a single AWS CodeBuild. It can stand on it's own for receiving trigger events from git services like AWS CodeCommit or Github. While AWS CodeBuild support executing build commands in stages, if you are looking for full-blown pipeline that supports various AWS or third party services, you can try AWS CodePipeline.

First thing we are going to discuss is the part createSource.

createSource(): codebuild.Source {
    const secret = cdk.SecretValue.secretsManager('GITHUB_OAUTH_TOKEN');
    new codebuild.GitHubSourceCredentials(this, 'GithubCredentials', {
        accessToken: secret,
    })
    const repo = ssm.StringParameter.valueForStringParameter(this, 'GITHUB_REPO');
    const owner = ssm.StringParameter.valueForStringParameter(this, 'GITHUB_OWNER');
    const source = codebuild.Source.gitHub({
        owner: owner,
        repo: repo,
        webhook: true,
        webhookFilters: [
            codebuild.FilterGroup.inEventOf(codebuild.EventAction.PUSH).andBranchIs('master'),
        ]
    });

    return source;
}

In this method, we are setting up the Source for the CodeBuild to fetch the source code. I am going to use GitHub as I hosted the source code there. Here I do configuration when creating the source object by calling codebuild.Source.gitHub. Before deploying the infrastructure you will need to set up some configurations so the CodeBuild will be able to integrate to your repository. You will need to obtain your GitHub OAuth Token and save it securely using AWS Secrets Manager.

aws secretsmanager create-secret \
    --name GITHUB_OAUTH_TOKEN \
    --secret-string abcdefg1234abcdefg56789abcdefg

You also need to store configuration for your repository by storing GITHUB_REPO and GITHUB_OWNER to AWS Systems Manager Parameter Store.

aws ssm put-parameter \
    --name GITHUB_OWNER \
    --type String \
    --value owner

aws ssm put-parameter \
    --name GITHUB_REPO \
    --type String \
    --value repo

The webhookFilters part specify that we only accept trigger event when a code is pushed to master branch.

Secondly is the simple part createProject. This will receive the source object we created before to create CodeBuild project.

createProject(source: codebuild.Source): codebuild.Project {
    return new codebuild.Project(this, 'Build', {
        source: source,
    });
}

And the third part is addTestReportPermissionToProject. Here we need to add permission for the CodeBuild to update the test report.

addTestReportPermissionToProject(project: codebuild.IProject) {
    //"arn:aws:codebuild:your-region:your-aws-account-id:report-group/my-project-*";
    const pattern = {
        partition: 'aws',
        service: 'codebuild',
        resource: `report-group/${project.projectName}-*`
    };
    const reportArn = cdk.Arn.format(pattern, cdk.Stack.of(this));

    project.addToRolePolicy(new iam.PolicyStatement({
        resources: [
            reportArn,
        ],
        effect: iam.Effect.ALLOW,
        actions: [
            "codebuild:CreateReportGroup",
            "codebuild:CreateReport",
            "codebuild:UpdateReport",
            "codebuild:BatchPutTestCases"
        ]
    }));
}

The test report resource ARN will use format arn:aws:codebuild:<your-region>:<your-aws-account-id>:report-group/<project-name>-*. We will use cdk.Arn.format helper to build the ARN for the IAM statement.

Deploying

If you already installed AWS CDK in your environment, you can go ahead with the following command

cdk deploy

It will show permission changes. Type y when you are prompted to proceed with the change.

Do you wish to deploy these changes (y/n)? y

It will then show progress on building the resources using CloudFormation.

Result

You now can see the created CodeBuild in your console. Here you will see the CodeBuild project that we created.

If you click on the project details, you will see the build history. As soon as you push a new code in your repository, a new entry will pop up.

The build entry will first show the build output from terminal execution.

You will be able to see the report history in the Reports tab after the build finished and the reports uploaded.

The summary of the history will show the pass rate and duration for the test executed in the run. The success will be shown in green and the failures will be shown in red.

You can also see the trend from past executions.

Clean Up

To avoid additional cost incurring, you may want to remove the resources by executing the following command.

cdk destroy

Summary and What's Next?

Now we can display reports from PHPUnit execution and track the trends by using AWS CodeBuild. We also know how to create CodeBuild and integrate to GitHub repo using AWS CDK. From here, there are couple of things that we can do to improve the developer experience and security.

Creating Pipeline The build system we created here is pretty simple. You might want to create more sophisticated pipeline using AWS CodePipeline, for example to send notification when the test fails.
Adding Permission to Developers IAM Group to Browse the List If you already have an IAM Group set up for your developer teams, you may want to add permissions to the group to browse and view the test reports.

Thanks a lot for reading this article. I would love to hear feedback or ideas from you. Feel free to drop your comments in the Comment section.

References

Here are some references that you can read to understand more about the Test Reports feature.

What Laravel programmers wants to know about AWS?

Petra Barus — Thu, 02 Apr 2020 04:08:56 +0000

I am currently developing a workshop series on building scalable Laravel application on AWS. This workshop will guide early-stage Laravel developers on starting on AWS, deploying first application, and introduce AWS services.

I have been a PHP programmer for more than a decade. While I mostly use Yii framework a long time (and contribute some of the source code), but during my activities with local PHP community, I met way more Laravel programmers than Yii2.

So I would like to know more about

1) What platform do you usually use (Lightsail, EC2, Beanstalk, ECS, EKS, Lambda)?

2) What are AWS services you usually integrate with?

3) What are some challenges you face currently or in the past?

Not sure how this discuss tag works, but hope this will reach out a lot of people.

Using Amazon DynamoDB for Session Storage in PHP/Yii2 Application (with AWS CDK)

Petra Barus — Sat, 28 Mar 2020 17:25:51 +0000

After we successfully deploy a PHP application in cluster environment, we still need some efforts to make the application to be stateless. One of them is building centralized session management. In this post, I will demonstrate how we can provision an Amazon DynamoDB table using AWS CDK and how to configure our application to securely use the created table for session storage. I will also demonstrate how to set up the local environment to test the session using Docker compose and Localstack.

This post is part of series Building Modern PHP/Yii2 Application using AWS (or see here for Indonesian version).

By default, PHP will store its session into local directory, usually in /tmp for Linux distributions. Since we are using load balanced multiple PHP applications, this can cause problem. We cannot really guarantee a single user will access the same application that will access the same file. Yes, we can set the Application Load Balancer to use Sticky Session so the load balancer can use cookie to target same instance for the same user. But consider when container stops because autoscaling suddenly scale down or case of the container fails. This will log users out who are still using that instance and cause bad experience.

To solve this, we are going to use DynamoDB as a centralized session storage. Since DynamoDB is a fully managed key-value database and it can delivers within single-digit miliseconds, it is a fitting solution for this challenge.

Using Session in Yii2

We are going to pick up from previous article where we create CI/CD Pipeline. You can access the code here.

In order to use session feature in Yii2, we will use predefined session from Yii application components. We can set a value into the component and the value will persist during the user's sessions.

The following code shows a controller where we are going to demonstrate using sessions.

Controller TestController will be able to receive query parameter name. If we pass a value with the query parameter name, it will save into the session. Otherwise it will get the previously set sessions. The controller will output the value stored in the session and the ID of the session.

After that we will enable routing so that we can access the controller through path /test/index. Add the following configuration to override the default urlManager component..

//file: /config/web.php
//put this in the `components` list of values.

//  'components' => [
//      .....
        'urlManager' => [
            'enablePrettyUrl' => true,
            'showScriptName' => false,
            'rules' => [
            ],
        ],

We can try deploying the new code using CDK command deploy. (I changed the stack name to MyWebStack to differentiate between the stack in the previous article).

cdk deploy MyWebStack

You can test by accessing URL <URL>/test/index, try setting the query parameter name using URL <URL>/test/index?name=foobar, and try the URL without query paramater. Note that sometimes the foobar shows up in the browser, but sometimes it does not.

DynamoDb Session Class

Fortunately for us, AWS SDK for PHP already provides an interface to use DynamoDB as session handler for PHP with class Aws\DynamoDb\SessionHandler. This class implements the core PHP interface SessionHandlerInterface. Yii2 also provides a nice binding with the SessionHandlerInterface with its class yii\web\Session which is the default class for session. We just need to pass the instance of DynamoDb SessionHandler to handler attribute of Yii2 Session class.

To install AWS SDK PHP composer dependencies, execute the following command.

composer require aws/aws-sdk-php

We then need to create a Session class to wrap the session access. The class will extends yii\web\Session, instantiate the SessionHandler from DynamoDB client, and pass it to handler.

Setting Up Local Environment

Now we are going to test this in the local environment. We are going to use Localstack docker image to mock DynamoDb. Localstack provides various mocks for AWS services, but here we are going to use only DynamoDb. This way, we don't need to test in the AWS environment that can take sometime to wait.

The DynamoDb will be accessible through url http://localstack:4569. We will set this in the environment variables alongside the Table name. We also need to provide fake AWS credentials and region in the environment.

environment:
  AWS_REGION: 'ap-southeast-1'
  AWS_ACCESS_KEY_ID: 'fake-access-key'
  AWS_SECRET_ACCESS_KEY: 'fake-secret-key'
  DYNAMODB_SESSION_TABLE_NAME: 'Sessions'
  DYNAMODB_ENDPOINT_URL: 'http://localstack:4569'

To initialize the DynamoDb table for storing sessions we will use the script below. The script will create a new DynamoDb table in the localstack. The table will be called Sessions and will have one hash key named id. It should be executed after the docker compose run.

Configuring Local Access

The following code will override the default session component to use our Session class. It will obtain the environment variables AWS_REGION, DYNAMODB_ENDPOINT_URL and DYNAMODB_SESSION_TABLE_NAME to the class so the class can pass it to the DynamoDB client and session handler.

//file: /config/web.php
//put this in the `components` list of values.

//  'components' => [
//      .....
        'session' => [
            'class' => app\components\Session::class,
            'clientConfigs' => [
                'version' => '2012-08-10',
                'region' => $_ENV['AWS_REGION'],
                'endpoint' => $_ENV['DYNAMODB_ENDPOINT_URL'] ?? null
            ],
            'sessionConfigs' => [
                'table_name' => $_ENV['DYNAMODB_SESSION_TABLE_NAME']
            ]
        ]

The final configuration will look like below.

Running in Local

We now have all the things we need to run the local environment. Execute this command to run the application.

docker-compose up -d

You may want to wait until all the containers complete the start up. Then execute this command to initialize the session table.

docker-compose exec localstack sh ./localstack-setup.sh

The site can be accessed in the http://localhost:8080. You can try the test controller by http://localhost:8080/test/index and http://localhost:8008/test/index?name=petra. Of course the session will be persistent since we only have 1 server to access.

Writing CDK Code

We are going to write the CDK code to provision DynamoDB table to store the sessions. But first we need to install the DynamoDB CDK module in the Node packages.

npm update
npm install @aws-cdk/aws-dynamodb

Here we only need to update the WebApp construct. The final code will look like this.

Now the code explanation.

The following code will provision a new Table with a single partition key called id and type String. This id attribute will store the session ID.

    private createSessionTable(): dynamodb.Table {
        return new dynamodb.Table(this, 'Sessions', {
            partitionKey: {
                name: 'id',
                type: dynamodb.AttributeType.STRING,
            }
        });
    }

After that, the following code will grant permission for the Fargate task's Task Role to update and write session to DynamoDB. We only need five permissions for this GetItem, UpdateItem, DeleteItem, Scan, and BatchWriteItem.

const actions = [
    "dynamodb:GetItem",
    "dynamodb:UpdateItem",
    "dynamodb:DeleteItem",
    "dynamodb:Scan",
    "dynamodb:BatchWriteItem"
]
this.sessionTable.grant(taskDefinition.taskRole, ...actions);

Finally, in the following code, we will pass through the environment variable AWS_REGION the region for the DynamoDB client to resolve and DYNAMO_SESSION_TABLE_NAME for the created DynamoDB table name.

taskImageOptions: {
    image: ecs.ContainerImage.fromAsset('.'),
    environment: {
        AWS_REGION: region,
        DYNAMODB_SESSION_TABLE_NAME: this.sessionTable.tableName,
    }
},

To apply this new resources and configurations, execute the following command.

cdk deploy MyWebStack

If you open the DynamoDB console, you can see the created new Table.

You can see the sessions are already filling up.

You can check the assigned value from the application by scanning the record ID of the row with the session ID. The image below shows that we have set name in the data attribute.

Wrapping Up

That concludes the tutorial on how to use DynamoDB for sessions storage. NOTE If you are still experimenting with this, please don't forget to destroy the resources by executing cdk destroy to avoid incurring cost.

I hope the explanation helps your development effort. I will remind you that this post is part of series Building Modern PHP/Yii2 Application using AWS. If there is any use cases that you would like to see me cover, please don't hesitate put it in the comments. Looking forward for the next post!

Building CI/CD Pipeline using AWS CodePipeline, AWS CodeBuild, Amazon ECR, Amazon ECS with AWS CDK

Petra Barus — Mon, 23 Mar 2020 17:02:35 +0000

This post is part of series Building Modern PHP/Yii2 Application using AWS. In this post, I will demonstrate how to build a CI/CD Pipeline for my code hosted at Github to deploy to our cluster at Amazon Elastic Container Service (ECS). I will use AWS CodePipeline, AWS CodeBuild, and Amazon Elastic Container Registry (ECR) with AWS Cloud Development Kit (CDK) to model the infrastructure.

In my personal experience, it is a good habit to set up Continuous Integration/Continuous Delivery (CD/CD) every time I set up a new project. This will reduce the amount of friction to test every commit in the production/staging environment. That way I can easily detect any code defects earlier, rather when it's too late. Building CI/CD pipeline can be very intimidating and it can have steep learning curve, so a lot of developers will do it at later stage of development. However by using Infrastructure as a Code, developers can easily configure the CI/CD pipeline and moreover they can use the template for next projects.

The set up will look like below. We are going to use AWS CodePipeline to model our pipeline. The pipeline itself will have 3 stages: Source, Build, and Deploy.

The first stage Source will fetch the source code from Github and send the artifact to the next stage. Second stage Build will use AWS CodeBuild to run the docker image building from the code artifact and then store the image to Amazon ECR. This stage will output the container name and. the image URL as artifact to next stage. The last stage Deploy will update Amazon ECS to use the new image URL for its container.

Setting Up Github Access

Before we start, we are going to set up access to our Github repository and store it to AWS Systems Manager Parameter Store and AWS Secrets Manager so our pipeline can access the code securely.

STEP 1 Go to Github Settings / Developer settings / Personal access token.

STEP 2 Choose Generate new token.

STEP 3 Fill in the name of the new token. And choose repo permission.

STEP 4 Choose Generate new token

Now we are going to store repository name and owner to the Parameter Store and access token to Secrets Manager. Execute the following commands, replace the owner, repo, and abcdefg1234abcdefg56789abcdefg with your configuration. You also can change the name with easily memorized parameter name.

aws ssm put-parameter \
    --name /myapp/dev/GITHUB_OWNER \
    --type String \
    --value owner

aws ssm put-parameter \
    --name /myapp/dev/GITHUB_REPO \
    --type String \
    --value repo

aws secretsmanager create-secret \
    --name /myapp/dev/GITHUB_TOKEN \
    --secret-string abcdefg1234abcdefg56789abcdefg

If you access the Parameter Store console, you can see that the parameter is now stored.

Similar if you access to the Secrets Manager console, you can see the secrets is stored.

Creating AWS CDK Code

Now we return back to our code from previous article. Or if you are new, you can access at my Github.

We then need to install couple of CDK dependencies for AWS CodePipeline, AWS CodeBuild, and Amazon ECR. Execute the following command.

npm update

npm install @aws-cdk/aws-codepipeline @aws-cdk/aws-codepipeline-actions @aws-cdk/aws-codebuild @aws-cdk/aws-ecr @aws-cdk/aws-ssm  @aws-cdk/aws-secretsmanager

We are going to refactor the existing WebStack into three Construct, Cluster, WebApp and Pipeline. See the following gist for the code.

https://gist.github.com/petrabarus/8accb7b61f895ff683f07819a2109b67

After this you need to execute following command to deploy.

cdk deploy MyWebAppStack

After deployment finished, you can check the CodePipeline console to see if the pipeline has been created.

Code Explanation

The best thing about AWS CDK that I love is, by using common programming language, I can still practice a lot of best programming best practices that I am already familiar with. With this, it is easier to read the code and explain the code to people.

Cluster Construct

The Cluster construct is fairly simple. It's just creating ECS Cluster. The instance of this construct will be passed to WebApp.

    readonly ecsCluster: ecs.Cluster;

    constructor(scope: cdk.Construct, id: string) {
        super(scope, id);
        this.ecsCluster = new ecs.Cluster(this, 'EcsCluster');
        this.output();
    }

At this point, there is not really a big reason why we need to pull out the ECS Cluster from the WebApp construct other than showing we can demonstrate using multiple construct. But it shows pretty good concept since an ECS cluster can have multiple services inside.

WebApp Construct

This construct is similar to the previous article. It is building a Fargate service (and setting up the autoscaling, since we left it there at previous article). However we refactor the ECS Cluster instantiation part to the Cluster Construct. We also introduce the ECR repository in this construct to store the image.

In the following line, we do three things. First we create the ECR repository, Secondly we give access to the Fargate task execution role to pull the docker image. And third, we get the defined service and container name. We then pass the ECR Repo, service, and container name to the pipeline.

    this.ecrRepo.grantPull(this.fargateService.taskDefinition.executionRole!);
        this.service = this.fargateService.service;
        this.containerName = this.fargateService.taskDefinition.defaultContainer!.containerName;

Pipeline Construct

The pipeline construct is a bit complex, however. We will need the WebApp instance for the constructor. From this we will obtain the ECS Service to update, the container to update, and the ECR repository to store the docker image. After that we call createPipeline to instantiate the CodePipeline Pipeline Construct.


interface PipelineProps {
    readonly webapp: WebApp;
}

class Pipeline extends cdk.Construct {
    private readonly webapp: WebApp;

    readonly service: ecs.IBaseService;
    readonly containerName: string;
    readonly ecrRepo: ecr.Repository;

    public readonly pipeline: codepipeline.Pipeline;

    constructor(scope: cdk.Construct, id: string, props: PipelineProps) {
        super(scope, id);
        this.webapp = props.webapp;
        this.service = this.webapp.service;
        this.ecrRepo = this.webapp.ecrRepo;
        this.containerName = this.webapp.containerName;

        this.pipeline = this.createPipeline();
        this.output();
    }

Pipeline

We create a new Pipeline construct with three stages and two artifacts. The stages are refactored into different methods to make the code more readable. The artifacts will be passed from stage to stage, e.g. the sourceOutput artifact will be output of Source stage and input to Build stage and buildOutput will be output of Build stage and input to Deploy stage.


    private createPipeline(): codepipeline.Pipeline {
        const sourceOutput = new codepipeline.Artifact();
        const buildOutput = new codepipeline.Artifact();
        return new codepipeline.Pipeline(this, 'Pipeline', {
            stages: [
                this.createSourceStage('Source', sourceOutput),
                this.createImageBuildStage('Build', sourceOutput, buildOutput),
                this.createDeployStage('Deploy', buildOutput),
            ]
        });
    }

Source Stage

In the Source stage, we create a GithubSourceAction to fetch the source securely from GitHub. Usually I use AWS CodeCommit as example, but this time I use GitHub because I am storing the code publicly there. To configure the source action, we will need to fetch the configurations that we had set up before. They are the GitHub repo and owner name we stored at Parameter Store and the GitHub token we stored in Secrets Manager. The output artifact will be the code fetched from the GitHub source.

private createSourceStage(stageName: string, output: codepipeline.Artifact): codepipeline.StageProps {
    const secret = cdk.SecretValue.secretsManager('/myapp/dev/GITHUB_TOKEN');
    const repo = ssm.StringParameter.valueForStringParameter(this, '/myapp/dev/GITHUB_REPO');
    const owner = ssm.StringParameter.valueForStringParameter(this, '/myapp/dev/GITHUB_OWNER');
    const githubAction = new codepipeline_actions.GitHubSourceAction({
        actionName: 'Github_Source',
        owner: owner,
        repo: repo,
        oauthToken: secret,
        output: output,
    });
    return {
        stageName: stageName,
        actions: [githubAction],
    };
}

Build Stage

Next stage, the Build stage will receive fetched GitHub code as input artifact and will send artifact output for the Deploy stage.

private createImageBuildStage(stageName: string, input: codepipeline.Artifact, output: codepipeline.Artifact): codepipeline.StageProps {

We created the CodeBuild in the following code. It receives the buildspec and environment definitions. Since we will use docker commands inside the build process, we need to set the privilegedMode as true. We also need to pass the ECR repository URL as environment variable as docker image push destination and container name to write the output artifact.

const project = new codebuild.PipelineProject(
    this,
    'Project',
    {
        buildSpec: this.createBuildSpec(),
        environment: {
            buildImage: codebuild.LinuxBuildImage.STANDARD_2_0,
            privileged: true,
        },
        environmentVariables: {
            REPOSITORY_URI: {value: this.ecrRepo.repositoryUri},
            CONTAINER_NAME: {value: this.containerName}
        }
    }
);

The following line will grant access for the Codebuild to pull or push image to ECR repository.

this.ecrRepo.grantPullPush(project.grantPrincipal);

Method createBuildSpec() will define the build specification for CodeBuild. But alternatively, we can set buildspec.yml in the project directory.

createBuildSpec(): codebuild.BuildSpec {

Following lines will set the runtime configuration and install the required NPM and composer. We need runtime nodejs and php to do both.

install: {
    'runtime-versions': {
        'nodejs': '10',
        'php': '7.3'
    },
    commands: [
        'npm install',
        'composer install',
    ],
},

In the following lines, we configure docker to access ECR and create tag name for tagging the docker image version we are going to build.

'aws --version',
'$(aws ecr get-login --region ${AWS_DEFAULT_REGION} --no-include-email |  sed \'s|https://||\')',
'COMMIT_HASH=$(echo $CODEBUILD_RESOLVED_SOURCE_VERSION | cut -c 1-7)',
'IMAGE_TAG=${COMMIT_HASH:=latest}'

The following lines will build the docker image, tag it, and push to ECR.

'docker build -t $REPOSITORY_URI:latest .',
'docker tag $REPOSITORY_URI:latest $REPOSITORY_URI:$IMAGE_TAG',
'docker push $REPOSITORY_URI:latest',
'docker push $REPOSITORY_URI:$IMAGE_TAG',

Finally, the last lines will generate a file named imagedefinitions.json in the root project that we call as Image Definitions File. It contains JSON with the container name and the repository URI pair. This file will be the output artifacts of the Build stage and passed to the Deploy stage.

            'printf "[{\\"name\\":\\"${CONTAINER_NAME}\\",\\"imageUri\\":\\"${REPOSITORY_URI}:latest\\"}]" > imagedefinitions.json'
        ]
    }
},
artifacts: {
    files: [
        'imagedefinitions.json'
    ]
}

Deploy Stage

The last stage, the Deploy stage will receive the imagedefinitions.json artifact and then deploy it to ECS using the ECSDeployAction.

    createDeployStage(stageName: string, input: codepipeline.Artifact): codepipeline.StageProps {
        const ecsDeployAction = new codepipeline_actions.EcsDeployAction({
            actionName: 'ECSDeploy_Action',
            input: input,
            service: this.service,
        });
        return {
            stageName: stageName,
            actions: [ecsDeployAction],
        }
    }

Main Application

The main CDK application now will contain WebStack that will simply call all the previous constructs.

    const cluster = new Cluster(this, 'Cluster');
    const webapp = new WebApp(this, 'WebApp', {
        cluster: cluster
    });
    const pipeline = new Pipeline(this, 'Pipeline', {
        webapp: webapp
    })

NOTE If you are still experimenting, don't forget to execute cdk destroy to avoid extra cost.

That is all for now. It is pretty long one. I hope you enjoy the explanation. Again, this post is part of series Building Modern PHP/Yii2 Application using AWS. If you have any interesting ideas, please post in the comment section. See you in the next post.

Building Modern PHP/Yii2 Application using AWS

Petra Barus — Thu, 19 Mar 2020 13:41:42 +0000

PHP has been the most popular programming language in Indonesia. I personally don't think it will change anytime soon. The language itself has been consistently adding a lot of new features these past few years, making programming in PHP safer and more structured. Very different face to the language I picked up in my high school years.

Among all popular PHP frameworks, it is by happenstance that I use Yii Framework as my main PHP framework for my startup. It was still running for 6.5 years when the company got acquired, although some of the components were already scraped and rewritten in other languages for performance and compatibility considerations. It was also happenstance that I chose AWS as my cloud provider for the company, and it was really great choice.

This series will discuss about how to build Modern PHP Yii2 application on AWS. These are some of best practices that I have borrowed from other people and worked well in my experience. With the fast growing number of services that AWS released, I will introduce new ideas that can help more developers using Yii2 to be more productive and agile.

The list below will keep expanding so don't forget to check it out from time to time. If you have any ideas or requests, please drop a comment :)

Adding AutoScaling to Amazon ECS with AWS CDK - Building Modern PHP/Yii2 Application using AWS

Petra Barus — Thu, 19 Mar 2020 13:14:18 +0000

In this section, I will show how to add autoscaling in the code that we have written in previous post.

Why do we need to use autoscaling? The reason is so that we can serve visitors to our site with optimal cost. Here is how it goes. The image below is the pattern of visitors to common e-commerce site like Amazon.com. Usually the visitor count peaks at day and goes down at night.

(Picture Credit: Scaling Up to Your First 10 Millions presentation)

In order to serve the visitors without any problem, we have to provision servers just enough for the peak performance.

With AWS AutoScaling, we can automatically turn off unused server capacility when the traffic goes down. Since we only pay the amount of servers that we turn on, we then can save a lot money.

So, coming back to our code in previous post, we use ApplicationLoadBalancedFargateService construct to create a site using Amazon ECS with AWS Fargate. This construct hides a lot of resources and process in the below layer.

const fargateService = new ecsPatterns.ApplicationLoadBalancedFargateService(this, 'Service', {
    cluster: ecsCluster,
    taskImageOptions: {
       image: ecs.ContainerImage.fromAsset('.')
    },
});

From the code above, we can configure the autoscaling with just a few line of code.

const fargateService = new ecsPatterns.ApplicationLoadBalancedFargateService(this, 'Service', {
    cluster: ecsCluster,
    taskImageOptions: {
        image: ecs.ContainerImage.fromAsset('.')
    },
});

const autoScalingGroup = fargateService.service.autoScaleTaskCount({
    minCapacity: 2,
    maxCapacity: 10
});

It can be seen from the code above that we take the AutoScalingGroup object by calling autoScaleTaskCount method and assign the object with minimum capacity of 2 and maximum capacity of 10. This means that in the peak time, the Fargate task will scale out up to 10 tasks. It will scale in to 2 task in off-peak time just in case one of the task go under failure.

Scaling By Schedule

We can configure the scaling by time schedule, means the capacity will go up or down based on the schedule that we set.

Untuk fitur penjadwalan ini kita harus mengimpor dependensi @aws-cdk/aws-applicationautoscaling di package.json. Eksekusi npm install untuk menginstall dependensi tersebut.

To enable the autoscaling, we need to install the CDK dependency @aws-cdk/aws-applicationautoscaling to our NPM package by executing the following command.

npm update
npm install @aws-cdk/aws-applicationautoscaling

Then in the app.ts we import the library.

import appAutoscaling = require("@aws-cdk/aws-applicationautoscaling");

We just need to call scaleOnSchedule method on the autoScalingGroup object. Usually the schedule done by adding task near the noon and removing task at night.

const autoScalingGroup = fargateService.service.autoScaleTaskCount({
    minCapacity: 2,
    maxCapacity: 10
});
autoScalingGroup.scaleOnSchedule('ScaleUpInMorning', {
    schedule: appAutoscaling.Schedule.cron({hour: '03', minute: '30'}),
    minCapacity: 10,
});
autoScalingGroup.scaleOnSchedule('ScaleDownInEvening', {
    schedule: appAutoscaling.Schedule.cron({hour: '10', minute: '00'}),
    maxCapacity: 5,
});

We schedule the task addition with minimum 10 task at 3.30 UTC time (it's 10.30 Jakarta time, almost noon). The task removal is scheduled at 10.00 UTC time with maximum 5 tasks (17.00 Jakarta time, afternoon).

To apply the change, execute the following command.

cdk deploy

Scaling By CPU Usage

In addition to configure autoscaling by schedule, we can also configure by other metrics such as CPU utilization. We can assume that the more visitors to our site, the more CPU usage will become.

The following code shows autoscaling will keep CPU utilization below 50%.

autoScalingGroup.scaleOnCpuUtilization('CpuScaling', {
    targetUtilizationPercent: 50,
    scaleInCooldown: cdk.Duration.seconds(60),
    scaleOutCooldown: cdk.Duration.seconds(60),
});

When the CPU utilization is below 50%, ECS will scale in to reduce the number of tasks, and when it reaches above 50%, it will add more task. The parameter scaleInCoolDown and scaleOutCoolDown is the length of time to wait before scale out and scale in process.

That is just a bit overview about auto scaling and how to add auto scaling capabilities to our existing PHP Yii2 application that we built before. Don't forget to execute cdk destroy to avoid additional cost if you are still exploring.

Please do drop comments if you have any questions or ideas for next post.

References

You can learn more about the autoscaling here

Deploying Yii2 Application to Amazon ECS and AWS Fargate using AWS CDK - Building Modern PHP/Yii2 Application using AWS

Petra Barus — Tue, 24 Dec 2019 11:00:02 +0000

This is my first tutorial posted at dev.to. I am currently building a series of tutorial on Building Modern PHP/Yii2 Application using AWS posted in my Personal Blog but in Indonesian language. Since I am now trying to improve my English, I will post the translated version here.

The series will follow a journey building a Yii2-based application from scratch to full application using AWS services. In this case we will use Amazon Elastic Container Service and AWS Fargate to host the application. We also will use AWS Cloud Development Kit to define the AWS resources we need to run the application. The application will follow some best practices and opinions from my personal experience.

I assume the readers already have AWS account and knows a bit about AWS, Docker, and PHP. Let's get started.

Here's the software that I used when building this tutorial

AWS CLI 1.16
PHP 7.3
Composer 1.19
Node 10
Node Package Manager 6.9
AWS CDK 1.19
Docker 19.03
Docker Compose 1.25

Installing Yii2 Basic App Template

In this series, I will build the Yii2 application using the yii2-app-basic template. As suggested by the name, it's a basic template that contains one single web application with small functionalities like home page, menu, login, and register.

To install the template, execute the command below.

composer create-project --prefer-dist yiisoft/yii2-app-basic myapp

We will now have a fresh Yii2 application project in myapp directory.

Let's try to run the application in local. In the myapp directory, execute the command below.

docker-compose up

This command will take a while to download the container image for running the application. It will then show logging line like this.

You can now access the application through URL http://127.0.0.1:8000. It should look like this.

Running AWS CDK

Adding CDK Script

To run the CDK you will need the scripts and some configurations. I already made a patch that you can apply on your project. Execute the command below.

curl -L -O http://bit.ly/modern-yii2-app_part-1_patch
patch -p1 < modern-yii2-app_part-1_patch

This patch will installs 9 files; 3 files for NodeJS and Typescript configuration, 3 files for Docker configuration, 1 .gitignore, and we have only 2 for the CDK itself.

Installing Requirements

After the scripts are patched, we now need to install Node libraries. Execute the command below.

npm install

To see if the CDK can be run, execute the command below.

cdk ls

The command will show a text WebStack. Up to this point, you are in the correct path. Please let me know if you have difficulty so far.

Because the current sample I wrote need a staging environment for the CDK itself to store data and configurations, we need to do bootstrapping first. The command below will try to create S3 bucket required for the CDK.

cdk bootstrap

It will show something like this.

Deployment

Now we try to deploy the application into AWS. You simply execute the command below.

cdk deploy

The command will show first changes of configuration that will be made, such as IAM and Security Group. It will ask you if you agree with the change or not. The configurations will affect the resources that you will provision, not existing in your account. Press y to continue.

We are now going to the deployment process. This process can take some time. It will first build the docker image locally and push to Amazon Elastic Container Repository. For Internet connection in Indonesia, this can take really while, that is why in the next series I will write about how to run the process in build server.

It then will provision the required resources like VPC, Subnets, ECS Clusters, and Load Balancer.

When it finished, the command will show the URL to the load balancer.

Open the URL and see that the Yii2 application is now deployed.

If we open the Amazon ECS console, we can see a new ECS cluster has been provisioned.

We can also see the service and task created after we opened the details for the cluster. It is displayed in the screenshot below that we now have 1 Fargate task to run the PHP application.

Redeploying

The command cdk deploy can also be executed again after we modified the code, say updating the PHP, HTML, JS, etc. Because the resources have been provisioned in the first deployment, the subsequent deployments will not take as much time unless you add new resources.

Deleting Resource

To avoid additional cost incurring, we need to destroy the resource if we don't need it again. We simply execute command below.

cdk destroy

Explanation

From all the files that we patched earlier, the one first one I like to explain is the app.ts. This Typescript file declares the infrastructure that we want to provision to run the application.

#!/usr/bin/env node
import 'source-map-support/register';
import cdk = require('@aws-cdk/core');
import ecs = require("@aws-cdk/aws-ecs");
import ecsPatterns = require("@aws-cdk/aws-ecs-patterns");

class WebStack extends cdk.Stack {
    constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {
        super(scope, id, props);
        this.createECSCluster();
    }

    createECSCluster() {
        const ecsCluster = new ecs.Cluster(this, 'ecsCluster');
        new ecsPatterns.ApplicationLoadBalancedFargateService(this, 'Service', {
            cluster: ecsCluster,
            taskImageOptions: {
                image: ecs.ContainerImage.fromAsset('.')
            },
        });
    }
}

const app = new cdk.App();
new WebStack(app, 'WebStack');

app.synth();

Today AWS CDK only supports Javascript/Typescript, Python, Java, and .NET. There is no support for PHP yet. But you want to see AWS CDK in PHP as much as I do, please send your vote in this Github issue.

With very simple code shown above we can have application with complete architecture similar to below.

Image taken from blog post Task Networking in AWS Fargate

Throughout the process, the code will be synthesized to CloudFormation template that can reach 700 lines like this. To see the generated CloudFormation template, execute the command below.

cdk synth

Construct

Now let us start with a simple line taken from app.ts.

const ecsCluster = new ecs.Cluster(this, 'ecsCluster');

The code above shows we are instantiating class ecs.Cluster. In AWS CDK, this is called as Construct. Construct is a building block that represents one or more components. It is useful to encapsulate multiple cloud components to be reusable as one unit. When building more complex architecture we can build a construct that composes multiple construct.

Stack

Next we will see the class WebStack.

export class WebStack extends cdk.Stack {
  constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);
    //....
  }
  //....
}

In this part, we define what we call as Stack. A stack is unit that defines the CloudFormation stack that we need to provisioned the resources. You can see in the CloudFormation console, that we have a new stack called WebStack, just like the one we instantiate.

App

In the last three lines, we instantiate what we call as App.

const app = new cdk.App();
new WebStack(app, 'WebStack');

app.synth();

The app will represent CDK application that we are building and defines the scope of the Stacks. An App can have more than one Stack. This is very useful to separate multiple parts of an application, for example the web stack and worker stack. In the code sample below we defines two stacks of which we name MyStack1 and MyStack2 respectively.

const app = new cdk.App();
new Stack1(app, 'MyStack1');
new Stack2(app, 'MyStack2');
app.synth();

If we execute cdk ls, we can see the two stacks will be displayed. It is also possible to deploy the two stacks in separate commands.

$cdk deploy MyStack1
$cdk deploy MyStack2

High Level Constructs

In the code app.ts, class ApplicationLoadBalancedFargateService is also a construct but represents very complex architecture. A construct can compose of many other construct. We can imagine that the ApplicationLoadBalancedFargateService can have other constructs for VPC, Subnet, Internet Gateway, NAT Gateway, ENI, Load Balancer, and Fargate. By using that construct, we don't have to worry about orchestrating all of the other constructs. We only need to instantiate the construct and give some parameters.

 const ecsCluster = new ecs.Cluster(this, 'ecsCluster');
        new ecsPatterns.ApplicationLoadBalancedFargateService(this, 'Service', {
            cluster: ecsCluster,
            taskDefinition: {
                cpu: 512,
                memoryLimitMiB: 1024
            },
            taskImageOptions: {
                image: ecs.ContainerImage.fromAsset('.')
            },
        });

The code above shows that we gives parameters to Fargate task specifying that we need CPU with 512 units and memory 1024 MiB.

Class ApplicationLoadBalancedFargateService is just an example of a construct already provided by aws-ecs-patterns library. The library also provides other patterns for ECS that we can choose. If we need EC2 for compute rather than Fargate, we can use ApplicationLoadBalancedEc2Service. Or if we choose to use Network Load Balancer instead of Application Load Balancer, because we are building games using UDP for protocol, we can use NetworkLoadBalancedFargateService. You can see the list of the patterns here.

Decomposing Using Construct

In the future we will definitely add more resource definition into the CDK code. Construct will help us to decompose the code so our code will stay simple and readable. The decomposition will further enable us for reusing the piece of code and even share it to other developers in form of libraries.

For example, the part in the app.ts where we define the ECS cluster and fargate can be refactor into new Construct, let's say WebECSCluster. We can also move the class to other file so our main file app.ts will stay short.

class WebECSCluster extends cdk.Construct {
    constructor(scope: cdk.Construct) {
        super(scope, "WebECSCluster");

        const ecsCluster = new ecs.Cluster(this, 'ecsCluster');
        new ecsPatterns.ApplicationLoadBalancedFargateService(this, 'Service', {
            cluster: ecsCluster,
            taskImageOptions: {
                image: ecs.ContainerImage.fromAsset('.')
            },
        });
    }
}


class WebStack extends cdk.Stack {
    constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {
        super(scope, id, props);
        new WebECSCluster(this);
    }
}

That is all for now. Please drop your comments or questions in the comment section below. I'd love to hear what you think I can improve in the next articles!

References

You can learn more about AWS CDK here