Arseny Zinchenko

Posted on Jun 20, 2019 • Originally published at rtfm.co.ua on Jun 6, 2019

Jenkins: running PHPUnit from Codeception by a Pull Request in Github and Allure-reports

#jenkins #git #unittesting #phpunit

The task is to create a Jenkins Scripted Pipeline job to run PHPUnit with our PHP-based backend unit-tests.

To run PHPUnit Codeception will be used.

This Jenkins job must be triggered from a Github repository after creating Pull Request so will use Github Pull-Request Builder plugin here.

To view generated reports – Allure Jenkins plugin will be used.

Our Jenkins is running in a Docker container and all its tasks also will be running as Docker containers using Docker plugin.

Testing tests

To get a picture how tests will be running in Jenkins – first, let’s run them locally.

PHP:

$ docker run -ti php:7.2 php --version
PHP 7.2.19 (cli) (built: Jun  1 2019 00:50:51) ( NTS )
Copyright (c) 1997-2018 The PHP Group
Zend Engine v3.2.0, Copyright (c) 1998-2018 Zend Technologies

Make init:

$ docker run -ti -v $(pwd):/data/ php:7.2 /data/init --env=Development
Yii Application Initialization Tool v1.0
...
initialization completed.

Now – Composer to install dependencies:

$ docker run -ti -v $(pwd):/data/ --workdir /data composer install --optimize-autoloader --ignore-platform-reqs
Loading composer repositories with package information
Installing dependencies (including require-dev) from lock file
Package operations: 118 installs, 0 updates, 0 removals
...
Generating optimized autoload files

And finally – unit-tests itself, from our backend directory first:

$ docker docker run -ti -v $(pwd):/data/ codeception/codeception run -c /data/backend/ unit
Codeception PHP Testing Framework v3.0.1
Powered by PHPUnit 7.5.9 by Sebastian Bergmann and contributors.
Running with seed:
Backend\tests.unit Tests (0) ----
Time: 99 ms, Memory: 16.00 MB
No tests executed!

Okay – there are no tests yet, developers will add them later and our task for now just make it works. Let’s go to Jenkins.

Jenkins

Create a Pipeline job, configure the GitHub project:

Write a pipeline script:

node {

    stage('Clone repo') {
        git branch: "master", url: "git@github.com:orgname/reponame.git", credentialsId: "jenkins-github"
    }

    stage('Build app') {
        docker.image('php:7.2').inside('-v /var/run/docker.sock:/var/run/docker.sock') {
            sh "php init --env=Development"
        }
    }
}

Run:

Okay, all work here, now add Composer:

node {

    stage('Clone repo') {
        git branch: "master", url: "git@github.com:orgname/reponame.git", credentialsId: "jenkins-github"
    }

    stage('Build app') {
        docker.image('php:7.2').inside('-v /var/run/docker.sock:/var/run/docker.sock') {
            sh "php init --env=Development"
        }
        docker.image('composer').inside('-v /var/run/docker.sock:/var/run/docker.sock') {
            sh "composer install --optimize-autoloader --ignore-platform-reqs"
        }
    }
}

Run the job and:

…

– Installing orgname/itunes-receipt-validator (dev-master c8c11ba): Cloning c8c11ba7ba

Failed to download orgname/itunes-receipt-validator from source: Failed to execute git clone –no-checkout ‘git@github.com:orgname/itunes-receipt-validator.git’ ‘/var/lib/jenkins/workspace/BackendUnitTests/ProjectName/vendor/orgname/itunes-receipt-validator’ && cd ‘/var/lib/jenkins/workspace/BackendUnitTests/ProjectName/vendor/orgname/itunes-receipt-validator’ && git remote add composer ‘git@github.com:orgname/itunes-receipt-validator.git’ && git fetch composer

Now trying to download from dist

– Installing orgname/itunes-receipt-validator (dev-master c8c11ba): Downloading (connecting…)… Downloading (failed)

[Composer\Downloader\TransportException]

The “https://api.github.com/repos/orgname/itunes-receipt-validator/zipball/c8c11ba7ba1741c7dec248e6f9c0787cb146730a” file could not be downloaded (HTTP/1.1 404 Not Found)

…

Add a Github Auth Token by calling composer config:

...
        docker.image('composer').inside('-v /var/run/docker.sock:/var/run/docker.sock') {
            sh "composer config -g github-oauth.github.com 64b***554"
            sh "composer install --optimize-autoloader --ignore-platform-reqs"
        }
...

Restart the job:

Nice.

Add tests execution:

node {

    stage('Clone repo') {
        git branch: "master", url: "git@github.com:orgname/reponame.git", credentialsId: "jenkins-github"
    }

    stage('Build app') {
        docker.image('php:7.2').inside('-v /var/run/docker.sock:/var/run/docker.sock') {
            sh "php init --env=Development"
        }
        docker.image('composer').inside('-v /var/run/docker.sock:/var/run/docker.sock') {
            sh "composer config -g github-oauth.github.com 64b***554"
            sh "composer install --optimize-autoloader --ignore-platform-reqs"
        }
    }

    stage('Backend tests') {
        docker.image('codeception/codeception').inside('-v /var/run/docker.sock:/var/run/docker.sock') {
            sh "codeception run -c backend/ unit"
        }        
    }
}

Jenkins Docker Plugin – The container started but didn’t run the expected command. Please double check your ENTRYPOINT.

The next error was when codeception Docker container was added:

…

[Pipeline] withDockerContainer

Jenkins seems to be running inside container 81f6a8cbdb28a86b5e156a929ef06c2a68dd5c716910f0ab66073656c32c6472

$ docker run -t -d -u 0:0 -v /var/run/docker.sock:/var/run/docker.sock -w /var/lib/jenkins/workspace/BackendUnitTests/ProjectName –volumes-from 81f6a8cbdb28a86b5e156a929ef06c2a68dd5c716910f0ab66073656c32c6472 -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** codeception/codeception cat

$ docker top 7e909bf25e232c951658478c1a82a93504061a4de4d90b2134680b3ca5f59f36 -eo pid,comm

ERROR: The container started but didn’t run the expected command. Please double check your ENTRYPOINT does execute the command passed as docker run argument, as required by official docker images (see https://github.com/docker-library/official-images#consistency for entrypoint consistency requirements).

Alternatively you can force image entrypoint to be disabled by adding option –entrypoint=”.

…

process apparently never started in /var/lib/jenkins/workspace/BackendUnitTests/ProjectName@tmp/durable-49ccb7e5

…

[Pipeline] End of Pipeline

ERROR: script returned exit code -2

Finished: FAILURE

It’s not the first time I see this error and usually, I’m solving it just by building own Docker image with all necessary tools.

But this time I was a bit lazy to write Dockerfile and build it, so found another “solution”.

In this Jenkins issue opened in 2016 and still with fresh comments, people say that --entrypoint="" may help.

But in this case the sh "codeception run [...]" call will not work.

Thus, need to find an executable Codeception in the Codeception Docker image.

This could be made by googling its Dockerfile with entrypoint specified, or by running a container and checking it with docker inspect but I did it in another way.

Run a container locally with redefined entrypoint to bash:

$ docker run -ti --entrypoint bash codeception/codeception
root@596960d14192:/project#

Then I tried to find codeception executable file:

root@596960d14192:/project# find / -name codeception
/root/.composer/cache/files/codeception
/repo/vendor/codeception

[/simterm]

But those all are directories…

Then how it is started?

So I started to type code and hit by the TAB button and it set me the executable name – codecept instead of codeception.

Then just call which codecept which returned me /repo/codecept.

Now update pipeline script – add --entrypoint="" and change sh call:

...
    stage('Backend tests') {
        docker.image('codeception/codeception').inside('-v /var/run/docker.sock:/var/run/docker.sock --entrypoint=""') {
            sh "/repo/codecept run -c backend/ unit"
        }        
    }
...

Run it:

Nice.

During writing this post till this lines – developers added a “test test” so let’s run it with reports generation, see docs here>>>.

...
    stage('Backend tests') {
        docker.image('codeception/codeception').inside('-v /var/run/docker.sock:/var/run/docker.sock --entrypoint=""') {
            sh "/repo/codecept run -c backend/ unit --coverage-xml --coverage-html"
        }        
    }
...

Run:

Allure reports

The next thing I want to do is an Allure report after the job will be finished.

We already have Allure Reports Plugin for another builds so let’s try to use it with Codeception/PHPUnit.

Another solution could be using Clover PHP Plugin.

Allure docs for Codeception is available here>>>.

Edit a project’s composer.json, add the Allure dependency:

...
{
    "require": {
            ...
      "allure-framework/allure-codeception": ">=1.1.0"
    }
}
...

Update backend/codeception.yml config and add extensions:

namespace: backend\tests
actor: Tester
paths:
    tests: tests
    log: tests/_output
    data: tests/_data
    helpers: tests/_support
settings:
    bootstrap: _bootstrap.php
    colors: true
    memory_limit: 1024M
modules:
    config:
        Yii2:
            configFile: 'config/test-local.php'
extensions: 
    enabled:
        - Yandex\Allure\Codeception\AllureCodeception
    config:
        Yandex\Allure\Codeception\AllureCodeception:
            deletePreviousResults: false
            outputDirectory: allure-results 
            ignoredAnnotations:
                - env        
                - dataprovider

In the job add composer.lock update (later better to update it in the project’s repository to avoid calling update each time in the job):

...
    stage('Build app') {
            ...
            sh "composer update --lock --ignore-platform-reqs"
            sh "composer install --optimize-autoloader --ignore-platform-reqs"
        }
    }
...

Add Allure plugin settings to the build script:

...
    stage('Backend tests') {
        docker.image('codeception/codeception').inside('-v /var/run/docker.sock:/var/run/docker.sock --entrypoint=""') {
            sh "/repo/codecept run -c backend/ unit"
        }        
    }

    stage('Reports') {
        allure([
            includeProperties: false,
            jdk: '',
            properties: [],
            reportBuildPolicy: 'ALWAYS',
            results: [[path: 'backend/tests/_output/allure-results']]
        ])
    }
...

Run the build:

And the report is ready:

Github webhook

The last thing to do is to configure a Github repository to execute a webhook after a Pull Request will be created to trigger unit tests job on the Jenkins.

See Jenkins: Github Pull-Request Builder плагин (Rus).

In the job’s settings add GitHub project:

Create a Github token and configure the plugin.

In the job’s settings in its Build Triggers enable GitHub Pull Request Builder, set Use github hooks for build triggering.

In the List of organizations. Their members will be whitelisted add a Github’s organization (or a dedicated user in the Admin list, if no organization in your Github), and set Allow members of whitelisted organizations as admins:

If a webhook wasn’t created by the plugin – add it yourself:

In the Payload URL set https:///ghprbhook/, chose Let me select individual events and select events which will trigger this webhook – Issue comments and Pull requests:

Create a PR:

Check the webhook status – must be a new call to the Jenkins:

And Jenkin’s job new build:

PR status:

In general – that’s all.

To finalize all we did above – move the Pipeline script to a repository and hardcoded Github repositories/branches – as the job’s parameters.

So the build-script now looks like this:

node {
    stage('Clone repo') {
        git branch: "${APPLICATION_BRANCH}", url: "${APPLICATION_URL}", credentialsId: "jenkins-github"
    }
    stage('Build app') {
        docker.image('php:7.2').inside('-v /var/run/docker.sock:/var/run/docker.sock') {
            sh "php init --env=Development"
        }
        docker.image('composer').inside('-v /var/run/docker.sock:/var/run/docker.sock') {
            sh "composer config -g github-oauth.github.com ${GITHUB_TOKEN}"
            sh "composer update --lock --ignore-platform-reqs"
            sh "composer install --optimize-autoloader --ignore-platform-reqs"
        }
    }
    try {
        stage('Backend tests') {
            docker.image('codeception/codeception').inside('-v /var/run/docker.sock:/var/run/docker.sock --entrypoint=""') {
                sh "/repo/codecept run -c backend/ unit"
            }
        }
        stage('Frontend tests') {
            docker.image('codeception/codeception').inside('-v /var/run/docker.sock:/var/run/docker.sock --entrypoint=""') {
                sh "/repo/codecept run -c frontend/ unit"
            }
        }
    } finally {
        stage('Reports') {
            allure([
                includeProperties: false,
                jdk: '',
                properties: [],
                reportBuildPolicy: 'ALWAYS',
                results: [[path: 'backend/tests/_output/allure-results'],[path: 'frontend/tests/_output/allure-results']]
            ])
        }
    }
}

There is a frontend tests stage added and tests execution set in the try/catch, to always collect reports by Allure (in the finally) otherwise, if any test will fail – build just will stop without executing the stage('Reports').

The script execution in the job now: