DEV Community: Brachi Packter

Integration test with GitHub Actions

Brachi Packter — Sun, 30 Aug 2020 19:37:17 +0000

Integration tests can run for hours, they cover edge cases and complex flows which the regular unit test suit doesn't check, in my case, integration test is deeper level of testing.

The point is that I don't want them to run for any commit in my PR.

Yes, this flow won't work for me:


name: Java CI 

on: [push]

jobs:
  build:

I want them to run before I make the merge, will the bellow flow satisfied me?

name: Java CI 

on:
  push:
    branches:
      - master

No, too late for me, the code is already merged, and if my tests will fail? master is broken? no way, I want to run them before the merge...

I came up with some "bot" style solution, I'll write a comment in the PR and it will trigger the integration job, it can be helpful to trigger it even before I'm ready to merge, just to get a feeling what is my PR status.

There are 2 issues with that solution (beside that it isn't automatic and require someone to "ask" for ci job..)

The job is not running in the PR context.
Need to check in any step that this is the required comment I looked for.

Both can be handled just keep it in mind, because it will make the solution a bit complex.

Let's start describe the flow step by step:

Here is the trigger, I want it to run per comment.

name: integration-test

on:
  issue_comment:
    types: [created]

But, only for comment with "ci" text :
I use a nice action that checks the comment text and give me output if that comment matches to my trigger word, it also has a nice ability to add emoji if the comment matches:

steps:
      - uses: khan/pull-request-comment-trigger@master
        id: check
        with:
          trigger: 'ci'
          reaction: rocket
        env:
          GITHUB_TOKEN: '${{ secrets.GITHUB_TOKEN }}'

After my comment is verified I want to send a message to the PR with the job link, that is needed because the build is not running in the PR context, it runs from the master, in order to make the linkage between the PR and the build, I send this comment back to the PR:

 - name: send comment 
        if: steps.check.outputs.triggered == 'true'
        env:
          URL: ${{ github.event.issue.comments_url }}
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        run: |
          curl \
            -X POST \
            $URL \
            -H "Content-Type: application/json" \
            -H "Authorization: token $GITHUB_TOKEN" \
            --data '{ "body": ":test_tube: [starting ci integration job]( https://github.com/'"$GITHUB_REPOSITORY"'/actions/runs/'"$GITHUB_RUN_ID"')" }'

The next step will pull the PR branch ,as already said, it is needed because the build isn't running in PR context

 - name: Get PR informations
        if: steps.check.outputs.triggered == 'true'
        id: pr_data
        run: |
          echo "::set-output name=branch::${{ fromJson(steps.request.outputs.data).head.ref }}"
          echo "::set-output name=repo_name::${{ fromJson(steps.request.outputs.data).head.repo.full_name }}"
          echo "::set-output name=repo_clone_url::${{ fromJson(steps.request.outputs.data).head.repo.clone_url }}"
          echo "::set-output name=repo_ssh_url::${{ fromJson(steps.request.outputs.data).head.repo.ssh_url }}"

- name: Checkout PR Branch
        if: steps.check.outputs.triggered == 'true'
        uses: actions/checkout@v2
        with:
          token: ${{ secrets.GITHUB_TOKEN }}
          repository: ${{ fromJson(steps.request.outputs.data).head.repo.full_name }}
          ref: ${{ steps.pr_data.outputs.branch }}

And, last thing, add the checked commit revision to the step output, I will need in next steps.

  - name: debug
        if: steps.check.outputs.triggered == 'true'
        id: debug
        run: |
          echo ::set-output name=sha::$( curl -u "u:${{github.token}}" https://api.github.com/repos/${{steps.pr_data.outputs.repo_name}}/git/ref/heads/${{steps.pr_data.outputs.branch}} | jq .object.sha | tr -d '"' )

Now comes the regular build step, in my case running Java unit test with Maven (-Pci will turn on maven profile that run only integration tests)

 - name: Set up JDK 1.8
        if: steps.check.outputs.triggered == 'true'
        uses: actions/setup-java@v1
        with:
          java-version: 1.8

      - name: Build with Maven
        if: steps.check.outputs.triggered == 'true'
        run: mvn clean install -Pci

After this step the job is failed or succeeded and I want to send a message indicates the status, it will send a link with the job build url by using ${{GITHUB_RUN_ID}}.

 - name: Create success comment
        if:  steps.check.outputs.triggered == 'true' &&  success()

        env:
          URL: ${{ github.event.issue.comments_url }}
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        run: |
          curl \
            -X POST \
            $URL \
            -H "Content-Type: application/json" \
            -H "Authorization: token $GITHUB_TOKEN" \
            --data '{ "body": ":v: [finish ci integration job successfully]( https://github.com/'"$GITHUB_REPOSITORY"'/actions/runs/'"$GITHUB_RUN_ID"')" }'    

      - name: Create fail comment
        if:  steps.check.outputs.triggered == 'true' && failure()

        env:
          URL: ${{ github.event.issue.comments_url }}
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        run: |
          curl \
            -X POST \
            $URL \
            -H "Content-Type: application/json" \
            -H "Authorization: token $GITHUB_TOKEN" \
            --data '{ "body": ":facepalm: [ci integration job failed]( https://github.com/'"$GITHUB_REPOSITORY"'/actions/runs/'"$GITHUB_RUN_ID"')" }'

Bonus part!
relevant if your build has Surefire reports
You can generate test report by using this nice action.
But you have to send it your last PR commit in order ti see the test results in the PR itself, this is the reason I output the commit sha id in the debug step.

      - name: Publish Test Report
        if: steps.check.outputs.triggered == 'true' && (failure() || success())
        uses: scacap/action-surefire-report@v1
        with:
          github_token: ${{ secrets.GITHUB_TOKEN }}
          commit: ${{ steps.debug.outputs.sha }}
          check_name: integration_test_results

That is!

Here is it how it looks in the PR conversation:

Intelij remote debugging java app in GKE (Google K8S Engine)

Brachi Packter — Fri, 28 Feb 2020 10:35:34 +0000

Prepare the docker image

In order to debug a java application we need to add some command line arguments to the remote app.

Set the arguments in the kubernetes deployment payload

DEBUG_OPTIONS="-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005"

Pass it to the java command in Dockerfile

ENTRYPOINT [ "sh", "-c", "java $DEBUG_OPTIONS -jar app.jar" ]

Expose the port 5005 in the GKE service

In GCP console, go to Kubernetes Engine -> Service & Ingress, choose your service and add the debug port:

ports:
  - name: http
    nodePort: 30060
    port: 80
    protocol: TCP
    targetPort: 8080
  - name: debug
    nodePort: 30080
    port: 5005
    protocol: TCP
    targetPort: 5005

Forward the pods ports to your localhost:5005

gcloud container clusters get-credentials ${CLUSTER_NAME} --region us-central1 --project ${PROJECT_NAME} \
 && kubectl port-forward --namespace ${NAMESPACE} $(kubectl get pod --namespace ${NAMESPACE} --selector="app=${APP}" --output jsonpath='{.items[0].metadata.name}') 5005:5005

You can generate this command from the google console by clicking on the button "port forwarding" near your debug port:

And now debug it as usual from Intelij

Lerna+Webpack+Serverless

Brachi Packter — Sun, 09 Feb 2020 22:03:01 +0000

In this Guide we will learn how to write Serverless services in Monorepo style.

When I tried to write a solutions combine Serverless, Webpack and Lerna together, I can't find any good guide how to do this, then, after implementing it I feel like I want to share my finding with the community,

What is Lerna?

Lerna,a tool for managing JavaScript projects with multiple packages, mostly used for Monorepo, Monorepo is beneficial strategy to keep all your repositories in a one global repository, contains all the common infrastructure and build tools, while each sub 'package' contains a different business project.

So if I have different services, I'll keep all of them in one repository, each one in a different package.

You have a file structure that looks like this:

  package.json
  packages/
    package-1/
      package.json
    package-2/
      package.json

My root package.json contains some common dependencies like aws-sdk and some dev-dependencies like webpack...

{
  "name": "root",
  "private": true,
  "dependencies": {
    "aws-sdk": "^2.531.0",
    "request-promise": "^4.2.5",
    "request": "^2.88.0"
  },
  "devDependencies": {
    "express": "4.17.1",
    "lerna": "^3.19.0",
    "serverless-webpack": "^5.3.1",
    "webpack": "^4.40.2",
    "webpack-cli": "^3.3.9"
  }
}

Read more about Lerna and it's feature in Lerna github.

Let's serverless it:

Serverless framework gives us the ability to deploy a serverless application (GCP,AWS,AZURE)

A basic example of yml file defining a Lambda:

service: MyService

provider:
  name: aws
  runtime: nodejs12.x

package:
  exclude:
    - node_modules/**

functions:
  myFunction:
    handler: index.myFunction

Let's bundle it, Adding webpack to the party:

We obviously won't want to deploy the full code to AWS, and we may want to use webpack to bundle it into one huge index file.

In order to do that I use serverless-webpack plugin

All the work needed, is just adding it to the serverless file:

service: MyService

provider:
  name: aws
  runtime: nodejs12.x

package:
  exclude:
    - node_modules/**

plugins:
  - serverless-webpack
functions:
  myFunction:
    handler: index.myFunction

And the index file wii look like this:

module.exports.myFunction = async (event, context) => {

    //doing all stuff needed :)
};

We mention that package is a business service unit, and it can be split to multiple Lambdas, for example UserService, can be split to 3 Lambdas for create, delete and update.

index.js will be like:

module.exports.createUser = async (event, context) => {

    //doing all stuff needed :)
};


module.exports.updateUser = async (event, context) => {

    //doing all stuff needed :)
};

module.exports.deleteUser = async (event, context) => {

    //doing all stuff needed :)
};

With its equivalent serverless.yml

service: UserService

provider:
  name: aws
  runtime: nodejs12.x

package:
  exclude:
    - node_modules/**

plugins:
  - serverless-webpack
functions:
  createUser:
    handler: index.createUser
  updateUser:
    handler: index.updateUser
  deleteUser:
    handler: index.deleteUser

What can I achieve from managing all my lambdas in the same repo?

Sure, I can have one yml file contains my VPC configuration, security groups, subnet ids, common environment variables and more, and all of them in a one root yml file. this can be referred from any serverless.yml:

provider:
  name: aws
  stage: prod
  runtime: nodejs12.x
  vpc:
    securityGroupIds: ${file(../../.root.yml):securityGroupIds}
    subnetIds: ${file(../../.root.yml):subnetIds}

  environment:
    ${file(../../.root.yml):environment}

And the root.yml:

securityGroupIds:
  - "sg-xxxxx"
subnetIds:
  - "subnet-xxxxx1"
  - "subnet-xxxxx2"
  - "subnet-xxxxx3"
environment:
  KEY1: value1
  KEY2: value2

SumUp

In this Guide we implement monorepo contains some services, each service can contains multiple lambdas, and each lambda is deployed using serverless framework, and we set some common configuration in the root folder to affect all sub packages.