DEV Community: AbdulBasit KABIR

Understanding Continuous Integration

AbdulBasit KABIR — Thu, 17 Aug 2017 22:18:54 +0000

Continuous Integration (CI) is a phase in the software development cycle where code from different team members or different features are integrated together. This usually involves merging code (integration), building the application and carrying out basic tests all within an ephemeral environment.

In the past, code was integrated at an “integration phase of the software development life-cycle. This phase came after different teams would have spent weeks, months or even years working in separately, dedicated to different (parts of) application or services. You can guess how painful the integration phase of the project would have been. It was not uncommon to spend weeks or even months in this phase. This was during the waterfall era.

With Extreme Programming (XP) and agile, integration became frequent with developers integrating as often as possible usually soon after a unit is complete. This is done on the shared source code repository. The frequent integration became automated and continuous which prompted the need for some kind of checks before the new code is integrated. Thus, Continuous Integration.

CI workflows vary a lot depending on tools, programming language, project and many other factors but a common flow has these steps.
• Pushing to the code repository
• Static analysis
• Pre-deployment testing
• Packaging and deployment to the test environment
• Post-deployment testing

Let’s take a look at how this flow would work on a java project with gradle as the build tool

Code repository

Typically there would have a code repository and some kind of workflow for contributing new code. Depending on the workflow, committing code kicks off the CI pipeline which often starts with static code analysis. You could refer to this post to help choose an appropriate workflow.

Static analysis

Static (code) analysis is done on the code base of the application without the need to run the software. The goal here is to ensure the code doesn’t have possible bugs and conforms to standard formatting and styling.

Let's add FindBugs to check for possible errors and Checkstyle to ensure the project conforms with coding standard (we’ll use the Google Java Style).
The Gradle build script would be something like this:

apply plugin: 'java'
apply plugin: 'findbugs'
apply plugin: 'checkstyle'


repositories {
    mavenCentral()
}

dependencies {
  ...
}

tasks.withType(FindBugs) {
  reports {
    xml.enabled false
    html.enabled true
  }
}

tasks.withType(Checkstyle) {
  reports {
    xml.enabled false
    html.enabled true
  }
}

To complete the Checkstyle configuration, we’ll need to add Checkstyle configuration file to the project path config/checkstyle/checkstyle.xml. There’s a sample config file on GitHub based on the Google Java style. With FindBugs and Checkstyle configured, static analysis can now be run:

$ gradle check

Pre-deployment testing

At this phase, any test that could be run without deploying to a server should be. This will include unit tests and various other types of tests (maybe functional or integration). This phase is used to ensure that the change set doesn’t break functionalities and works well with other parts of the code since the test are run on the whole codebase not just the new changes (as the author might have done on the dev env).

Once this phase of the ci pipeline passes successfully, the new changes are good to be accepted and ready to be deployed for further test (automated and user testing if needed). However, code review by other team members between this phase and the next would also improve the overall quality of the code as they would be able to spot a misunderstood requirement.

Packaging and deployment to the test/staging environment

Depending on the kind of project, the application is built, packaged, sent to a test or staging environment (that mimics production). This ensures that the integrated changes build well with other parts and can be deployed for functional test can to be carried out. It’s at this phase that we also verify that the new changes are compatible with other libraries and the deployment environment. This is phase should also automated.

Post-deployment testing

For tests that need the application to be deployed, this phase of the CI pipeline is where they’re run. The tests vary depending on tools, frameworks and language of the application but they’re usually functional integration and performance tests. Successful execution of this phase ends the CI pipeline for the changeset signalling it’s good enough for users.

Once the CI pipeline completes successfully, the deployed application could undergo manual test by a “user or the QA team to ensure that it fits the client’s requirements. The packages or artifacts generated by the CI pipeline can now be taken/deployed to production. This can also be automated with a successful implementation of Continuous Delivery (CD) pipeline.

A successful implementation of a CI/CD pipeline helps ensure quality and improves speed and take us a step closer to continuous delivery. Most source code management tools (like Github, Bitbucket, Gitlab) either have a CI pipeline add-on or can easily be integrated with other CI/build tools (like Jenkins, Travis CI, CircleCI). My personal preference is to integrate with Jenkins.
Hope you find this useful. look forward to seeing how this is implemented for different projects.

Which is the right git workflow for my next project?

AbdulBasit KABIR — Thu, 11 May 2017 20:47:16 +0000

So you want to adopt git to version your current or next project. There are quite a number of ways you to go about it and even collaborate with others. Apart from using git to just version the code base of your project, it could be used to coordinate how others contribute to the project. This is best achieved by adopting a workflow.

A workflow is not a concrete rule that you must follow but rather a guideline.

A git workflow is a collaborative practice to make contributing to any project easier and better organised. It outlines how to go about adding and publishing changes to a project’s source code repository. It’s most effective for projects that have a remote repository. A workflow is not a concrete rule that you must follow but rather a guideline. So feel free to pick aspects of one and combine with that of another to suit your need.

For the sake of this post, we’ll assume that master is your main branch and that you are familiar with basic git related terms (branch, commit, merge, pull request, patch, fork, repository) or you can google them.

Here are a few workflows to consider for your project. For each workflow, we’ll briefly look at how it works and where it’ll most likely fit. If there are requests, we could go into more details in a later post.

Central Workflow

The repository contains only one branch, the main branch, master. All changes are committed to this branch. The repository could be just local, with out a remote copy or also hosted remotely where it could be cloned and pushed to.

Use case

This is ideal for a one person project. On your project, you would adopt this workflow to have snapshots of your changes as you go along with the development. After some significant work, you commit your changes so that you can later go back to any previous version. It could also be used by small teams making the transition from svn to git.

Developer Branch Workflow

Every developer has their “personal” branch or subset of it, to which they push. All changes published to the remote repository by that developer would be on their branch. Work can be done on separate branches but would have to be published on the developer’s branch which is later integrated(merged) into the main branch.

Use case

Better suited for small(limited requirements) projects with a small number of collaborators needing their changes to be reviewed before it’s merged into master. Say you have a group assignment, collaborators on that team do their part then publish to the remote repository for others to review before it’s merged. The review should ideally be done with a pull(merge) request. It could also be a handy way to introduce pull request on a team or in an organisation.

Feature Branch Workflow

In its simplest form, the repository would have a main branch with stable, shippable code and other branches for every feature(or bug or improvement) to be integrated into the main branch. To go further, the repository would have a secondary main branch (eg dev) which holds stable code being tested to be shipped to users when it’s merged to master. In that case, feature branches are merged to dev, not master. There’s a very detailed explanation by Vincent Driessen.

Use case

This would be more appropriate for teams having some kind of project management method(e.g Agile) that continuously ship out. Let’s say your project is under continuous development and you might have a set of features to be developed for the next release. These features are assigned to different developers who create a branch for every feature that's published (most likely for review) before it’s merged into dev for testing. When you’re ready to release, dev is merged to master.

Issue Branch Workflow

Very similar to feature branch workflow with one key difference. Branches are created from issues/tasks on the issues in the project’s issue tracker. The branches could have the same name of the issue of its id. There’s only one branch per issue and one issue per branch.

Use case

Just Like the feature workflow, this is best suited for projects with a project management structure in place. However, unlike it, this workflow is more applicable to projects where each feature isn’t completely handled by a single developer. Like you would work on the UI of a feature and another developer would work on a different aspect of that feature. It could be adopted by both projects that are continuously being released or those that are released once in a while.

Forking Workflow

With this workflow, contributions to a project are made by creating a fork of its repository. All changes are committed to any branch on the forked repository then contributed back to the original repository with a pull request. Contributors would only have read access to the remote repository.

Use case

Mostly used on open source projects with public repositories. Anybody (let’s say a friend) that can view your remote repository can make a fork of it. They mustn’t have access to contribute directly to the repository as long as they can view it. When they’re done working on it, they make a pull request which you can review to decide whether to integrate, reject or ask for improvement before it’s merged into your project.

Patch Workflow

Using this workflow, contributors submit changes they’ve made to the repository with a patch (a file containing what was changed in the repository). The patch is applied to the repository by someone who can write directly to the repository, like a maintainer/owner.

Use case

Used on projects where the contributor can’t write directly to the repository but has access to the source code. Let’s say you share the source code of your project with a friend or they have read access your remote repository. After changes are committed to their copy to the source code, a patch is created and sent to you. You apply it to the repository to get their changes. Also used on some open source projects by those that aren’t main contributors of the project.

There could be tonnes of other way people collaborate using git or even different implementations/names for the ones discussed here. These are just a few. It’s left for you to choose any, combine multiple or something entirely different for your next project. So feel free to pick and choose.

I’d Love to see what you build and how you do it.