DEV Community: Juan Alonso

Proper Magento 2 Composer Setup

Juan Alonso — Mon, 26 Mar 2018 13:51:11 +0000

This post was originally posted at my colleague's Blog https://blog.hauri.me/

Goals

When thinking about how to setup your Magento 2 projects, you should have the following goals:

Simple repository structure
Only the project files under version control
Easily identify on local installations which files belong to the repo and which ones are installed by composer

These goals are very important in order to keep the repository light weight, well organised and simple for developers to know which files are under VCS.

That said, we will want something like in this image.

We will also expect the same result locally after installing all external dependencies and the Magento 2 project files.

Wrong Magento Docs

If you simple follow the Magento Docs, they will tell you to run:

composer require magento/project-community-edition

This configuration alone is not enough because you end up with something like that locally.

As you can see, there are lots of Magento files installed in your project root. Only the ones in the red are comming from the project repository. This solution is not valid for us because developers cannot clearly see which files are under version control and which ones where installed by composer. .gitignore is also quite complex as you need to ignore several individual files installed by composer out of the vendor folder.

Proper Configuration

In order to improve that, we take a slighty different approach:

1- Enclose all Magento files and dependencies under a specific folder, so they do not pollute your project root directory. In our case, we decided to call that folder magento.

For that, you only need to add magento/ prefix path in a couple of places of your composer.json:

    "config": {
        "vendor-dir": "magento/vendor"
    },
    "extra": {
        "magento-root-dir": "magento/"
    },
    "autoload": {
        "psr-4": {
            "Magento\\Setup\\": "magento/setup/src/Magento/Setup/"
        },
        "psr-0": {
            "": "magento/app/code/"
        },
        "files": [
            "magento/app/etc/NonComposerComponentRegistration.php"
        ]
    },
    "autoload-dev": {
            "psr-4": {
            "Magento\\Sniffs\\": "magento/dev/tests/static/framework/Magento/Sniffs/",
            "Magento\\Tools\\": "magento/dev/tools/Magento/Tools/",
            "Magento\\Tools\\Sanity\\": "magento/dev/build/publication/sanity/Magento/Tools/Sanity/",
            "Magento\\TestFramework\\Inspection\\": "magento/dev/tests/static/framework/Magento/TestFramework/Inspection/",
            "Magento\\TestFramework\\Utility\\": "magento/dev/tests/static/framework/Magento/TestFramework/Utility/"
            }
      }

2- .gitignore is also much simpler now. You only need to ignore the magento folder except your project specific files inside magento/app and your magento/app/etc/config.php:

    # Ignore all Magento code files except our code in "magento/app"
    /magento/*
    !/magento/app
    /magento/app/etc/*
    !/magento/app/etc/config.php
    /magento/app/code/Magento
    /magento/app/design/*/Magento
    /magento/app/i18n/magento
    /magento/app/*.*

3- Symlink composer.json and composer.lock files inside magento folder. For some strange reasons, Magento console needs the composer files under the same directory as the Magento root files. With this setup that is no longer the case, so you need to symlink these files after the composer install/update commands:

    "scripts": {
        "symlinkComposerFilesToMagentoRoot": "cd magento && ln -sf ../composer.* .",
        "post-install-cmd": [
            "@symlinkComposerFilesToMagentoRoot"
        ],
        "post-update-cmd": [
            "@symlinkComposerFilesToMagentoRoot"
        ]
    }

4- Just mention that now your server DocumentRoot configuration must point to <project_path>/magento/pub

Result

Now, if you execute composer install in a clean folder you should get all composer files installed under the magento folder.

Great! isn't it? Now the local installation looks much cleaner and exactly the same as our repository.

Extra tip:

If you want to be smarter, you could use a tool called symlinker-pro for adding your symlinks after composer install/update. Using this tool has following advantages:

Symlinks configuration centralized in one file
Possible to overwritte Magento default config files with your project specific configuration. For example:
- magento/dev/integration/phpunit.xml
- magento/dev/unit/phpunit.xml
- magento/dev/tools/grunt/configs/themes.js

In our case, we configure this tool to symlink every file inside our symlinks folder to the original magento directory.

Troubleshooting

Trying to install Magento sample data returns an error with this setup.

Problem:

sampledata:deploy command changes the composer --working-dir into the magento path.

Workaround:

Remove '--working-dir' => $baseDir on \Magento\SampleData\Console\Command\SampleDataDeployCommand::82 before executing sampledata:deploy

Summary and Files Overview

After applying these configurations our setup and files will look like that:

Project Structure

magento.symlinks

composer.json=>magento/composer.json
composer.lock=>magento/composer.lock

symlinks/magento/**=>magento

composer.json

{
  "name": "project/magento2-project-name",
  "description": "Magento Project Name",
  "type": "project",
  "license": [
    "proprietary"
  ],
  "authors": [
    {
      "name": "Your name",
      "email": "your.name@domain.com"
    }
  ],
  "config": {
    "vendor-dir": "magento/vendor",
    "use-include-path": true
  },
  "repositories": [
    {"type": "composer", "url": "https://repo.magento.com/"}
  ],
  "require": {
    "n98/magerun2": "^1.4",
    "magento/product-community-edition": "*",
    "composer/composer": "@alpha",
    "staempfli/symlinker-pro": "~1.0"
  },
  "require-dev": {
    "phpunit/phpunit": "^4.1"
  },
  "extra": {
    "magento-root-dir": "magento/",
    "magento-force": "override"
  },
  "autoload": {
    "psr-4": {
      "Magento\\Setup\\": "magento/setup/src/Magento/Setup/"
    },
    "psr-0": {
      "": "magento/app/code/"
    },
    "files": [
      "magento/app/etc/NonComposerComponentRegistration.php"
    ]
  },
  "autoload-dev": {
    "psr-4": {
      "Magento\\Sniffs\\": "magento/dev/tests/static/framework/Magento/Sniffs/",
      "Magento\\Tools\\": "magento/dev/tools/Magento/Tools/",
      "Magento\\Tools\\Sanity\\": "magento/dev/build/publication/sanity/Magento/Tools/Sanity/",
      "Magento\\TestFramework\\Inspection\\": "magento/dev/tests/static/framework/Magento/TestFramework/Inspection/",
      "Magento\\TestFramework\\Utility\\": "magento/dev/tests/static/framework/Magento/TestFramework/Utility/"
    }
  },
  "scripts": {
    "preventRunningComposerInMagentoFolder": "[ ! $(basename `pwd`) = 'magento' ] || { echo 'Checking current dir is not magento folder'; exit 1; }",
    "setPermissionsBin": "chmod -R +x bin magento/vendor/bin",
    "setSymlinks": "magento/vendor/bin/symlinker-pro create:from:file config/symlinks/magento.symlinks -f --enable-wildcards",
    "pre-install-cmd": [
        "@preventRunningComposerInMagentoFolder"
    ],
    "pre-update-cmd": [
        "@preventRunningComposerInMagentoFolder"
    ],
    "post-install-cmd": [
        "@setPermissionsBin",
        "@setSymlinks"
    ],
    "post-update-cmd": [
        "@setPermissionsBin",
        "@setSymlinks"
    ]
  }
}

.gitignore

.idea
.DS_Store

# Ignore all Magento code files except our code in "magento/app"
/magento/*
!/magento/app
/magento/app/etc/*
!/magento/app/etc/config.php
/magento/app/code/Magento
/magento/app/design/*/Magento
/magento/app/i18n/magento
/magento/app/*.*

Magento 2 Custom Database Connection - The Right Way

Juan Alonso — Mon, 26 Mar 2018 13:40:01 +0000

This post was originally posted at my colleague's Blog https://blog.hauri.me/

Adding a custom database connection in Magento2 seems easy at first but doing it properly it is a bit more complicated. Here I will explain how to do it right.

Normally, if you want to add a new database connection in your Magento 2 project, you will tend to do only the following. That is at least what I also did in the past 😅

1- Edit app/etc/env.php with new database connection

    'db' => array (
        'connection' => array (
            //...
            'custom' => array (
                'host' => '<host>',
                'dbname' => '<database>',
                'username' => '<user>',
                'password' => '<password>',
                'active' => '1',
            ),
        ),
   ),
        'resource' => array(
        //...
        'custom' => array(
          'connection' => 'custom'
        )
   )

2- Use the new connection in your model/resource, or wherever you want, by specifying the custom resource connection:

In your Model Resources

    public function __construct(Context $context, $connectionName = 'custom')
    {
        parent::__construct($context, $connectionName);
    }

At any other place

    /** @var Magento\Framework\App\ResourceConnection $this->resourceConnection **/
    $connection = $this->resourceConnection->getConnection('custom');

Quite easy, right? After that, you get your module working with the new database connection. However, if you want to do it the right way, that is not enough. Otherwise, why would we write this post?

If you only do that, you will face 2 issues:

This new database connection will not be avaiable for integration tests
Clean installations will crash if your module contains a setup script that uses this database. For example if you create/alter tables on this database using setup scripts.

The solution to overcome these problems is to add the configuration automatically into app/etc/env.php, the same way as the default_setup is added by default in Magento2. That way, our custom database configuration will always be available for clean installations and integration tests.

Good news is that Magento 2 is already prepared to allow other modules to extend the default setup:install command, in order to execute custom actions on setup time. The key is in Magento\Setup\Model\ConfigOptionsListCollector::collectOptionsLists

/**
 * Auto discover ConfigOptionsList class and collect them.
 * These classes should reside in <module>/Setup directories.
 */
public function collectOptionsLists()
{
    $optionsList = [];

    // go through modules
    foreach ($this->fullModuleList->getNames() as $moduleName) {
        $optionsClassName = str_replace('_', '\\', $moduleName) . '\Setup\ConfigOptionsList';
        if (class_exists($optionsClassName)) {
            $optionsClass = $this->objectManagerProvider->get()->create($optionsClassName);
            if ($optionsClass instanceof ConfigOptionsListInterface) {
                $optionsList[$moduleName] = $optionsClass;
            }
        }
    }
}

As you can see on previous function, this collectOptionsLists loads all module classes with namespace \Setup\ConfigOptionsList. Knowing that, we need to create our own ConfigOptionsList with the logic to create our custom database configuration. Our class will be as follows:

Vendorname\Modulename\SetupConfigOptionsList

<?php
/**
 * ConfigOptionsList
 *
 */
namespace Vendorname\Modulename\Setup;

use Magento\Framework\Config\Data\ConfigData;
use Magento\Framework\Config\File\ConfigFilePool;
use Magento\Framework\Setup\ConfigOptionsListInterface;
use Magento\Framework\Setup\Option\TextConfigOption;
use Magento\Framework\App\DeploymentConfig;
use Magento\Framework\Config\ConfigOptionsListConstants;
use Magento\Setup\Exception as SetupException;
use Magento\Framework\Model\ResourceModel\Type\Db\ConnectionFactory;

class ConfigOptionsList implements ConfigOptionsListInterface
{
    const DB_CONNECTION_NAME = 'custom';
    const DB_CONNECTION_SETUP = self::DB_CONNECTION_NAME;
    const CONFIG_PATH_DB_CONNECTION = ConfigOptionsListConstants::CONFIG_PATH_DB_CONNECTIONS . '/' . self::DB_CONNECTION_NAME;
    const CONFIG_PATH_RESOURCE_SETUP = ConfigOptionsListConstants::CONFIG_PATH_RESOURCE . '/' . self::DB_CONNECTION_SETUP . '/connection';
    const OPTION_DB_HOST = self::DB_CONNECTION_NAME . '-db-host';
    const OPTION_DB_NAME = self::DB_CONNECTION_NAME . '-db-name';
    const OPTION_DB_USER = self::DB_CONNECTION_NAME . '-db-user';
    const OPTION_DB_PASSWORD = self::DB_CONNECTION_NAME . '-db-password';
    const OPTION_DB_ENGINE = self::DB_CONNECTION_NAME . '-db-engine';
    const OPTION_DB_INIT_STATEMENTS = self::DB_CONNECTION_NAME . '-db-init-statements';

    const OPTIONAL_OPTIONS = [
        ConfigOptionsListConstants::KEY_HOST => self::OPTION_DB_HOST,
        ConfigOptionsListConstants::KEY_NAME => self::OPTION_DB_NAME,
        ConfigOptionsListConstants::KEY_USER => self::OPTION_DB_USER,
        ConfigOptionsListConstants::KEY_PASSWORD => self::OPTION_DB_PASSWORD,
        ConfigOptionsListConstants::KEY_ENGINE => self::OPTION_DB_ENGINE,
        ConfigOptionsListConstants::KEY_INIT_STATEMENTS => self::OPTION_DB_INIT_STATEMENTS,
    ];

    protected $connectionFactory;

    public function __construct(
        ConnectionFactory $connectionFactory
    ) {
        $this->connectionFactory = $connectionFactory;
    }

    public function getOptions()
    {
        return [
            new TextConfigOption(
                self::OPTION_DB_HOST,
                TextConfigOption::FRONTEND_WIZARD_TEXT,
                self::CONFIG_PATH_DB_CONNECTION . '/' . ConfigOptionsListConstants::KEY_HOST,
                'Database host (Connection name: Import)',
                'localhost'
            ),
            new TextConfigOption(
                self::OPTION_DB_NAME,
                TextConfigOption::FRONTEND_WIZARD_TEXT,
                self::CONFIG_PATH_DB_CONNECTION . '/' . ConfigOptionsListConstants::KEY_NAME,
                'Database name (Connection name: Import)',
                'magento2_import'
            ),
            new TextConfigOption(
                self::OPTION_DB_USER,
                TextConfigOption::FRONTEND_WIZARD_TEXT,
                self::CONFIG_PATH_DB_CONNECTION . '/' . ConfigOptionsListConstants::KEY_USER,
                'Database username (Connection name: Import)',
                'root'
            ),
            new TextConfigOption(
                self::OPTION_DB_PASSWORD,
                TextConfigOption::FRONTEND_WIZARD_PASSWORD,
                self::CONFIG_PATH_DB_CONNECTION . '/' . ConfigOptionsListConstants::KEY_PASSWORD,
                'Database password (Connection name: Import)',
                ''
            ),
            new TextConfigOption(
                self::OPTION_DB_ENGINE,
                TextConfigOption::FRONTEND_WIZARD_TEXT,
                self::CONFIG_PATH_DB_CONNECTION . '/' . ConfigOptionsListConstants::KEY_ENGINE,
                'Database engine (Connection name: Import)',
                'innodb'
            ),
            new TextConfigOption(
                self::OPTION_DB_INIT_STATEMENTS,
                TextConfigOption::FRONTEND_WIZARD_TEXT,
                self::CONFIG_PATH_DB_CONNECTION . '/' . ConfigOptionsListConstants::KEY_INIT_STATEMENTS,
                'Database initial set of commands (Connection name: Import)',
                'SET NAMES utf8;'
            ),
        ];
    }

    public function createConfig(array $options, DeploymentConfig $deploymentConfig)
    {
        $configData = new ConfigData(ConfigFilePool::APP_ENV);

        $dbConfig = $this->getDbConfig($options, $deploymentConfig);
        foreach ($dbConfig as $configSubPath => $configValue) {
            $configData->set(self::CONFIG_PATH_DB_CONNECTION . '/' . $configSubPath, $configValue);
        }

        if ($deploymentConfig->get(self::CONFIG_PATH_RESOURCE_SETUP) === null) {
            $configData->set(self::CONFIG_PATH_RESOURCE_SETUP, self::DB_CONNECTION_NAME);
        }

        return [$configData];
    }

    protected function getDbConfig(array $options, DeploymentConfig $deploymentConfig) : array
    {
        $dbConfig = [];
        foreach (self::OPTIONAL_OPTIONS as $configSubPath => $option) {
            if ($options[$option] === null) {
                $options[$option] = $deploymentConfig->get(self::CONFIG_PATH_DB_CONNECTION . '/' . $configSubPath);
            }
            $dbConfig[$configSubPath] = $options[$option];
        }

        $activeConfigPath = self::CONFIG_PATH_DB_CONNECTION . '/' . ConfigOptionsListConstants::KEY_ACTIVE;
        $dbConfig[ConfigOptionsListConstants::KEY_ACTIVE] = $deploymentConfig->get($activeConfigPath)??'1';

        return $dbConfig;
    }

    public function validate(array $options, DeploymentConfig $deploymentConfig)
    {
        if (!$options[ConfigOptionsListConstants::INPUT_KEY_SKIP_DB_VALIDATION]) {
            try {
                $dbConfig = $this->getDbConfig($options, $deploymentConfig);
                $this->checkDatabaseConnection($dbConfig);
            } catch (\Exception $e) {
                $errors = [
                    sprintf('Error validating DB connection name: "%s"', self::DB_CONNECTION_NAME),
                    $e->getMessage()
                ];
                return $errors;
            }
        }
        return [];
    }

    /**
     * @param array $dbConfig
     * @return bool
     * @throws SetupException
     */
    protected function checkDatabaseConnection(array $dbConfig)
    {
        $connection = $this->connectionFactory->create($dbConfig);
        $dbName = $dbConfig[ConfigOptionsListConstants::KEY_NAME];

        $accessibleDbs = $connection
            ->query("SHOW DATABASES")
            ->fetchAll(\PDO::FETCH_COLUMN, 0); //@codingStandardsIgnoreLine
        if (in_array($dbName, $accessibleDbs)) {
            return true;
        }

        throw new SetupException(
            sprintf("Database '%s' does not exist or user does not have privileges to access this database.", $dbName)
        );
    }
}

Let's break down what we are doing here:

1- Our class must implement ConfigOptionsListInterface. Checking this interface, we see that we have to implement 3 methods:

    interface ConfigOptionsListInterface
    {
        /**
         * Gets a list of input options so that user can provide required
         * information that will be used in deployment config file
         */
        public function getOptions();

        /**
         * Creates array of ConfigData objects from user input data.
         * Data in these objects will be stored in array form in deployment config file.
         */
        public function createConfig(array $options, DeploymentConfig $deploymentConfig);

        /**
         * Validates user input option values and returns error messages
         */
        public function validate(array $options, DeploymentConfig $deploymentConfig);
    }

2- Implement method getOptions(). Here is where we specify new options that will be available when executing install:setup. For a new database connection we will add following options:

    --custom-db-host                Database host (Connection name: Custom)
    --custom-db-name                Database name (Connection name: Custom)
    --custom-db-user                Database username (Connection name: Custom)
    --custom-db-password            Database password (Connection name: Custom)
    --custom-db-engine              Database engine (Connection name: Custom)
    --custom-db-init-statements     Database initial set of commands (Connection name: Custom)

Each option must be type TextConfigOption as in the following example:

    public function getOptions()
    {
        return [
            new TextConfigOption(
                self::OPTION_DB_HOST,
                TextConfigOption::FRONTEND_WIZARD_TEXT,
                self::CONFIG_PATH_DB_CONNECTION . '/' . ConfigOptionsListConstants::KEY_HOST,
                'Database host (Connection name: Import)',
                'localhost'
            ),
       //...
       ];
    }

3- Implement method validate(). Here we check database settings by trying to connect to the custom database. Validation will return errors if connection could not be established.

    public function validate(array $options, DeploymentConfig $deploymentConfig)
    {
        if (!$options[ConfigOptionsListConstants::INPUT_KEY_SKIP_DB_VALIDATION]) {
            try {
                $dbConfig = $this->getDbConfig($options, $deploymentConfig);
                $this->checkDatabaseConnection($dbConfig);
            } catch (\Exception $e) {
                $errors = [
                    sprintf('Error validating DB connection name: "%s"', self::DB_CONNECTION_NAME),
                    $e->getMessage()
                ];
                return $errors;
            }
        }
        return [];
    }

4- Implement method createConfig(). Finally the method that we were looking for. Here it is where we save the custom database settings into the app/etc/env.php configuration file:

    public function createConfig(array $options, DeploymentConfig $deploymentConfig)
    {
        $configData = new ConfigData(ConfigFilePool::APP_ENV);

        $dbConfig = $this->getDbConfig($options, $deploymentConfig);
        foreach ($dbConfig as $configSubPath => $configValue) {
            $configData->set(self::CONFIG_PATH_DB_CONNECTION . '/' . $configSubPath, $configValue);
        }

        if ($deploymentConfig->get(self::CONFIG_PATH_RESOURCE_SETUP) === null) {
            $configData->set(self::CONFIG_PATH_RESOURCE_SETUP, self::DB_CONNECTION_NAME);
        }

        return [$configData];
    }

And we are done! Clean installations and integration tests will now include our custom database settings automatically during setup.

Executing setup:install --help will show the new command options. The same way, if we do a new installation, we'll see that the custom database configuration gets automatically added into the app/etc/env.php.

This has the following advantages:

You can now add setup scripts that affect your custom database without crashing clean installations. env.php is now edited during the setup:install, so your custom database configuration is ready before the setup scripts are executed.
You can run integration tests that use your custom database. For that, you need to edit dev/tests/integration/etc/install-config-mysql.php with new options

  return [
        //...
        'custom-db-host' => '<host>',
        'custom-db-user' => '<user>',
        'custom-db-password' => '<password>',
        'custom-db-name' => '<custom_database>_integration_tests',
    ];

One last tip. If you want to skip the hassle of creating this code manually, you can use the magento2-code-generation tool to generate it automatically:

mg2-codegen template:generate customDBConnection

That's all. I hope this will help you when you add your custom databases. It was not so easy for me to find out the right way to do it but it was great to see that Magento2 is properly prepared for it.

Centralise Jenkins Pipelines configuration using Shared Libraries

Juan Alonso — Mon, 15 May 2017 16:06:59 +0000

This post is a continuation of the tutorial Setup Continuos Integration/Delivery system in just 4 steps with Jenkins Pipelines and Blue Ocean.

Now that you know how to setup your pipelines, you might want to go one step further and keep just one pipeline configuration for all your projects. That is possible using Shared Libraries. The idea is that your standard pipeline configuration resides in a shared repository that will be accessed for each of your projects. Each project will then only set specific properties into its own Jenkinsfile.

Let’s see how to do it:

Shared Library Repository

1- Create a new git repository where your shared library will reside.
2- Create a vars directory and a .groovy file inside. Call this file standardPipeline.groovy for example.
3- Add the following code example into this file.

    def call(body) {

        def config = [:]
        body.resolveStrategy = Closure.DELEGATE_FIRST
        body.delegate = config
        body()

        node {
            // Clean workspace before doing anything
            deleteDir()

            try {
                stage ('Clone') {
                    checkout scm
                }
                stage ('Build') {
                    sh "echo 'building ${config.projectName} ...'"
                }
                stage ('Tests') {
                    parallel 'static': {
                        sh "echo 'shell scripts to run static tests...'"
                    },
                    'unit': {
                        sh "echo 'shell scripts to run unit tests...'"
                    },
                    'integration': {
                        sh "echo 'shell scripts to run integration tests...'"
                    }
                }
                stage ('Deploy') {
                    sh "echo 'deploying to server ${config.serverDomain}...'"
                }
            } catch (err) {
                currentBuild.result = 'FAILED'
                throw err
            }
        }
    }

4- Commit and push

Jenkins configuration

1- Go to Manage Jenkins > Configure System > Global Pipeline Libraries
2- Give a name to your library and set the git url

Use library in your Project

1- Go to your project repository root and edit the content of your Jenkinsfile with just the following

    @Library("your-library-name") _
    standardPipeline {
        projectName = "Project1"
        serverDomain = "Project1 Server Domain"
    }

2- Note that the name of the library must be the same as you previously set in Jenkins.

That's it! You can now check in Blue Ocean how your projects are using the shared pipeline configuration

Review

Before we finish, let's see more in detail the code we used:

@library("your-library-name") _

Here is where you define which library to import. You can use as many libraries as you want into the same Jenkinsfile. Do not forget the _ at the end, it is needed!

standardPipeline {
    //...
}

The name of this block has to match with the name of your .groovy library name. In this example that would be standardBuild because our file was called standardPipeline.groovy

standardPipeline {
    projectName = "Project1"
    serverDomain = "Project1 Server Domain"
}

projectName and serverDomain are an example of how to set properties that can be accessed on the shared library:

def call(body) {
    def config = [:]
    body.resolveStrategy = Closure.DELEGATE_FIRST
    body.delegate = config
    body()

    //...

This code is responsible to load the properties previously mentioned. Thanks to that, properties can be accessed using config.propertyName

You can use properties to identify project settings or to execute different actions. For example:

if (config.propertyName == true) {
    // Do something here   
}

That's all. Now you can play around and configure your shared pipelines. We use 2 pipelines configurations:

projectBuild.groovy for our projects
commonModuleBuild.groovy to build common modules that can be installed as project dependencies.

commonModuleBuild.groovy is a bit different as it does not need a Deploy step for example.

You can find the resources used for this tutorial here:

Setup Continuos Integration/Delivery system in just 4 steps with Jenkins Pipelines and Blue Ocean

Juan Alonso — Sun, 07 May 2017 09:29:32 +0000

Now that the first stable version of Blue Ocean Plugin was released, it is easier and cooler than ever to setup your Continuous Integration/Delivery with Jenkins. Jenkins has also evolved quite a lot in recent years, making easier to setup your automation server and builds in just a couple of steps.

First of all, let’s clarify some of the concepts that we will be using in this tutorial:

Multibranch Pipelines: Pipelines, and especially multibranch pipelines, are a game changer in Jenkins. Thanks to this plugin, you can simply setup your repository url and Jenkins will identify all your branches. It will also automatically start new builds as soon as new commits are pushed, so no more webhooks or cumbersome configuration is needed.

Jenkinsfile: Together with Pipelines the new concept of Jenkinsfile was introduced. This is a file that you create in your repo, which contains your pipeline configuration. Jenkins will then look for this file in your branches and execute the build according to the stages defined in there. That makes possible to have your pipeline configuration together with your project and under version control.

Blue Ocean: This plugin is must for everyone using jenkins pipelines. Blue Ocean it is an opensource plugin that rethinks the user experience of Jenkins. Most amazing feature is the beautiful user interface of Pipelines, allowing for fast and intuitive comprehension of build's status. If the awful user experience of Jenkins was holding you back from using it, there are not more reasons for that.

Ok, so now that these concepts are clear, let's see how we can get our Continuous Delivery environment up and running in just 4 steps:

1. Jenkins installation:

You can follow the installation steps for your OS or Docker in the official docs:

https://jenkins.io/doc/book/getting-started/installing/

For this post, let's assume that you are using Ubuntu. You can log in into your system as sudo and execute the following commands:

wget -q -O - https://pkg.jenkins.io/debian/jenkins-ci.org.key | sudo apt-key add -
sudo sh -c 'echo deb http://pkg.jenkins.io/debian-stable binary/ > /etc/apt/sources.list.d/jenkins.list'
sudo apt-get update
sudo apt-get install jenkins

This automatically creates a jenkins user and daemon that listens on port 8080. That means that your Jenkins system is now reachable on http://<your_ip>:8080. In case that you are testing that locally, that would be http://127.0.0.1:8080

If you do not know your server public IP, you can execute ifconfig to get this information.

Open the url and follow the installation wizard. Here we recommend that you to select the default option to install the suggested plugins.

2. Intall Plugins

Go to Manage Jenkins > Manage Plugins > Available and filter by Blue Ocean.
Select and Install
If you use Bitbucket for your repository, you must also install Bitbucket Branch Source Plugin

3. Jenkinsfile

In your project repository root, create a new file called Jenkinsfile. This file is written in Groovy and will define your Pipeline Stages configuration. For this tutorial, just copy the following dummy example and paste it into the Jenkinsfile just created.
Save it, commit and push.

node {
    // Clean workspace before doing anything
    deleteDir()

    try {
        stage ('Clone') {
            checkout scm
        }
        stage ('Build') {
            sh "echo 'shell scripts to build project...'"
        }
        stage ('Tests') {
            parallel 'static': {
                sh "echo 'shell scripts to run static tests...'"
            },
            'unit': {
                sh "echo 'shell scripts to run unit tests...'"
            },
            'integration': {
                sh "echo 'shell scripts to run integration tests...'"
            }
        }
        stage ('Deploy') {
            sh "echo 'shell scripts to deploy to server...'"
        }
    } catch (err) {
        currentBuild.result = 'FAILED'
        throw err
    }
}

4. Setup Multibranch Pipeline

Go to the jenkins home page and click on new Item. Give a name to your Job and select Multibranch Pipeline.
- if you use Bitbucket, you must select Bitbucket Team/Project option instead.
After that, we need to configure the repository url and credentials. You can see examples of that in the following screenshots depending on your source control system (Bitbucket, github, git):
- Very important here is that you select Jenkinsfile as Build Configuration Mode.
Click save. You will notice that Jenkins starts scanning your repo in search for all your branches. In fact, what Jenkins is doing, it is to look for Jenkinsfiles inside your branches.
Click on the header Blue Ocean button.
et voilÃ ! you just setup a Continuous Delivery system that will automatically scan, build and deploy all your branches in your repository.
Now click around on the Blue Ocean pipelines branches to see how awesome it looks.

As you might have noticed, our Jenkinsfile is just defining the stages and printing some echos. The idea is that you replace these echos with your actual scripts. You can see a real example of what we are doing for our Magento 2 builds at the following link:

Jenkinsfile real example

You can also find more info about Jenkinsfile options and syntax here:

https://jenkins.io/doc/book/pipeline/jenkinsfile/

In a follow up post, I will explain how you can go one step further and keep only one pipeline configuration for all your project in a shared repository. Stay tuned!