DEV Community: Asdrubal Santander

Gitlab CI/CD Multi-Project Pipeline

Asdrubal Santander — Sun, 26 Jul 2020 20:53:46 +0000

Gitlab CI/CD

Gitlab CI/CD is the continuous integration and continuous delivery solution created by Gitlab, if you need help understanding the basics please refer to other sources like:

Multi-Project Pipeline

Is a feature that allows us to trigger pipelines that are hosted in other projects.

This is especially useful for microservices projects that need a way to deploy everything in the correct order, for projects that need to run E2E tests before deploying to production, or any other case that you can think of that needs integrated pipelines between projects.

Launching another's project pipeline was initially introduced to Gitlab in the form of API Triggers. Triggers are excellent when you need to launch a pipeline outside of Gitlab or for another project that you or your organization do not own. But the use of this type of triggers needs additional configurations like create a trigger token or use curl inside the job script.

Multi-Project Pipeline became free for all users in early 2020, but without the visualization of the pipeline triggered, like this:

In this case, we are going to explain some of the configurations in the Multi-Project Pipeline feature and from a free account ;)

Use Case:

Let's assume the following:

We are developing an awesome app by our development team.
The development team has already done some functionality of the awesome app and is already deployed to production.
But we also have a QA team that needs to make E2E testing to our awesome app.

Projects

my-awesome-project: This is the project with the code we want to deploy to production.
e2e-testing-qa: This is the project we want to trigger before the deployment phase in my-awesome-project, the main purpose of this project is to do E2E testing in a staging environment.

* This projects reside inside a group in Gitlab called my-awesome-group-trigger

Pipelines

Current pipeline flow:

In the current flow we have a stage to unit-testing, build our Docker image to later be deployed to production. We need to find a way to integrate this flow of the development team with the QA team and add some E2E tests.

First desired pipeline flow:

So this is a problem because the code related to our E2E tests would be in the same project that our production code, and this involves the two teams. So probably the best approach, in this case, is to have separate projects, this means we need to use a strategy to launch and sync our deployment pipelines.

Final pipeline flow:

This would integrate the two projects and only when all the previous stages to deploy are successfully passed the code would be launched to production.

* I would explain later what this Bridge job is.

CI/CD without the integration

my-awesome-project/.gitlab-ci.yml

image: alpine

stages:
    - test
    - build
    - deploy

test:
    stage: test
    script: 
        - echo "This is the test job."

build:
    stage: build
    script: 
        - echo "This is the build job."

deploy:
    stage: deploy
    script: 
        - echo "This is the deploy job."

e2e-testing-qa/.gitlab-ci.yml

image: alpine

stages:
    - deploy
    - e2e

deploy:
    stage: deploy
    script: 
        - echo "This is the deploy job."

e2e:
    stage: e2e
    script:         
        - echo "This is the E2E job."

As you can see we just need to make the QA integration, for this we are going to use the Multi-Project pipeline.

Bridge Job:

This is a job that has the special keyword trigger and this automatically turns this job in a bridge job, this means that its only purpose is to launch a pipeline in another project or the same project (see parent-child-pipelines).

Integration

In this particular case we only need to modify the CI/CD file in my-awesome-project, let's add the bridge stage and job, between build and deploy to make our E2E testing:

# my-awesome-project/.gitlab-ci.yml

image: alpine

stages:
    - test
    - build
    - bridge
    - deploy

test:
    stage: test
    script: 
        - echo "This is the test job."

build:
    stage: build
    script: 
        - echo "This is the build job."

bridge:
    stage: bridge
    trigger: my-awesome-group-trigger/e2e-testing-qa

deploy:
    stage: deploy
    script: 
        - echo "This is the deploy job."

Let's see how our pipelines are working together (for the following images the top pipeline is the project my-awesome-project and the bottom is e2e-testing-qa)

Passing pipeline

Falling pipeline

Works! But it seems like the pipeline in the project my-awesome-project it's not waiting for the e2e-testing-qa to finish, and this is not our desired behavior for this case, remember we need to make the E2E tests before the production deployment. So we need to modify our bridge job one more time to add the strategy keyword:

# my-awesome-project/.gitlab-ci.yml

...

bridge:
    stage: bridge
    trigger: my-awesome-group-trigger/e2e-testing-qa
    strategy: depend

...

Passing pipeline

Falling pipeline

Finally, when the E2E testing has passed in the e2e-testing-qa our deploy to production can be made automatically, and when it fails will also be falling our main pipeline.

Setting up Pytest for a Django project

Asdrubal Santander — Sat, 07 Dec 2019 21:40:04 +0000

Hi, my name is Asdrubal Santander, I'm a Backend developer at Mo Tecnologías, and this is a short guide to how we implement testing with Pytest-Django.

Why Pytest?

Pytest it is an alternative framework to write tests in Python besides Unittest, between its best features it has:

Easy to write assert statements just like any other conditional assert 1 == 1. No more assertNotNull, assertDictEquals...
Fixtures to run code before and after each test, set up state of a test, load fixtures and more!
Better integration with plugins and tools (like pdb, and coverage).
Improved results, when a test fail it tells you exactly where it fails and with colors!
Can run tests written with the syntax of Unittest. So it makes easy to change.

Our Project structure:

project_name
├── project
│   ├── apps
│   │   ├── integration_apps
│   │   |   ├── app_name
|   |   |   |   └── tests
|   |   |   |       ├── component_to_be_tested
|   |   |   |       |   ├── test_function_name.py
|   |   |   |       |
|   |   |   |       ├── conftest.py
|   |   |   |       └── ...
|   |   |   └── ...
|   |   └── ...
│   ├── project_settings
│   |   └── settings
|   |       ├── development_settings
|   |       ├── production_settings
|   |       └── test_settings
|   └── manage.py
├── pytest.ini
└── ...

So as you can see, our Django project as an uncommon structure, I will try to explain in a general way:

project_name: config files, requirements folder, README.MD ...
project: apps, fixtures folder, project_settings and manage.py.
project_settings: settings, project urls.py, wsgi.py.
settings: settings for our project in this case we are going to be focus on test_settings.
apps: contains our business logic in separated apps.
test: inside this folder we have separated conftest.py to an specific app, and is where we are going to put all our tests, separated by folders such like "views", "services", "selectors"...

Installation:

pip install pytest-django
pip install pytest-cov
pip install pytest-pythonpath

pytest-django: A plugin to easily integrate Pytest with Django.
pytest-cov: Plugin that helps to create the coverage reports inside Pytest.
pytest-pythonpath: Plugin that modifies the env var PYTHONPATH. This is useful when pytest can't find your manage.py

pytest.ini:

[pytest]
addopts = -v -p no:warnings --nomigrations --cov=. --no-cov-on-fail
DJANGO_SETTINGS_MODULE = project.project_settings.settings.test_settings
python_paths = .

With the file pytest.ini we can set arguments and other configurations, so we don't have to specify each time we run Pytest.

addopts: Special variable to set command line options
- -v: To tell pytest to be a little bit more verbose
- -p no:warnings: Disable warnings.
- --nomigrations: Disable the migrations when run pytest this could make the setup for testing faster.
- --cov=.: When pytest runs it makes a coverage report.
- --no-cov-on-fail: Don't show the coverage report if a test fails.
DJANGO_SETTINGS_MODULE: This is where we tell pytest to run with a specific settings.
python_paths: The dot meaning that the current directory will be added to the PYTHONPATH env var.

Running Pytest:

$ pytest 

======================================== test session starts ========================================
platform darwin -- Python 3.6.0, pytest-5.2.2, py-1.8.0, pluggy-0.13.0 -- 
...
cachedir: .pytest_cache
Django settings: project.project_settings.settings.test_settings (from ini file)
rootdir: .../mysite, inifile: pytest.ini
plugins: django-3.7.0, pythonpath-0.7.3, cov-2.8.1
collected 0 items



---------- coverage: platform darwin, python 3.6.0-final-0 -----------
Name                                                                Stmts   Miss  Cover
---------------------------------------------------------------------------------------
...
mo_services/apps/importantapp/test/services/test_function_name.py       0      0   100%
...
--------------------------------------------------------------------------------------------
TOTAL                                                                   X     X    XXX%


======================================= no tests ran in 0.09s =======================================

Let's write a example test:

Inside of app_name we create a python package named tests, this package will have all the tests for this application divided in sub packages with the name of the component:

# .../apps/integration_apps/app_name/tests/component_to_be_tested/test_function_name.py

from app_name.component_name import some_function


class TestFunctionName:

    def test_example():
        result = some_function()

        expected_result = 'some_result'

        assert result == expected_result

Running pytest again:

$ pytest 
======================================== test session starts ========================================
platform darwin -- Python 3.6.0, pytest-5.2.2, py-1.8.0, pluggy-0.13.0 -- 
...
cachedir: .pytest_cache
Django settings: project.project_settings.settings.test_settings (from ini file)
rootdir: .../mysite, inifile: pytest.ini
plugins: django-3.7.0, pythonpath-0.7.3, cov-2.8.1
collected 1 item

apps/integration_apps/app_name/tests/component_to_be_tested/test_function_name.py::TestFunctionName::test_example PASSED [100%]

--------------- coverage: platform darwin, python 3.6.0-final-0 -----------
Name                                                                Stmts   Miss  Cover
--------------------------------------------------------------------------------------------
...
mo_services/apps/importantapp/test/services/test_function_name.py       1      0   100%
...
--------------------------------------------------------------------------------------------
TOTAL                                                                   X     X    XXX%

========================================= 1 passed in 0.11s =========================================

So now you have successfully run Pytest in Django!

Some final cool command line options:

-x: This option tells pytest to stop after the first test fail, is great when we are having problems with several tests so in this way we can solve this problems one by one.
--nf: Tells pytest to run the "new tests" first.
--ff: Tells pytest to run the "fail tests" first.
--lf: Tells pytest to run the only the "last fail".