Automating CI/CD Pipelines with Bitbucket Webhooks and Jenkins: A Dockerized Demo

Rahul Malhotra — Fri, 27 Sep 2024 05:56:04 +0000

Introduction
In the fast-paced development world, automation in Continuous Integration (CI) and Continuous Deployment (CD) pipelines is crucial. Integrating Bitbucket's PR system with Jenkins allows automatic build triggering whenever developers comment on the pull requests. This blog showcases a Proof of Concept (PoC) that demonstrates how to automate multibranch pipeline triggers using Docker containers, Bitbucket webhooks, and Jenkins.

Who Is This Relevant For?:

Developers using Bitbucket and Jenkins for CI/CD
Teams managing multibranch pipelines and looking to automate the build process based on pull request comments

To demonstrate how a Dockerized environment for Jenkins and Bitbucket can be integrated using webhooks, allowing multibranch pipelines to be automatically triggered whenever a pull request is opened or commented upon in Bitbucket we will be using the following

Tools & Setup:

Docker Compose – Used to deploy Jenkins and Bitbucket in isolated containers.
Bitbucket Webhooks – To detect PR events and trigger the appropriate Jenkins job.
Jenkins Multibranch Pipelines – To automate builds based on the branch and repository.
Python Flask Application – A webhook listener that parses incoming Bitbucket PR events and triggers the respective Jenkins pipeline using Jenkins API.

For demonstration, we have dockerized the example, into following folder structure

project-root/
│
├── docker-compose.yaml
│
├── webhook-listener/
│   ├── Dockerfile
│   ├── app.py
│   └── requirements.txt
│
└── README.md

docker-compose.yaml:
The file that defines and orchestrates the services (Jenkins, Bitbucket, and the webhook listener).
webhook-listener:
Contains the Flask webhook listener code, a Dockerfile to create the image for the webhook app, and a requirements.txt to list the Python dependencies.

Step-by-Step Implementation

Docker Compose Setup The docker-compose.yaml sets up all services in a single file. This allows you to bring up Jenkins, Bitbucket, and the Flask app with one command.

version: '3.8'

services:
  jenkins:
    image: jenkins/jenkins:lts-jdk11
    container_name: jenkins
    restart: always
    ports:
      - "8080:8080"
      - "50000:50000"
    volumes:
      - jenkins_home:/var/jenkins_home
      - /var/run/docker.sock:/var/run/docker.sock  # To allow Jenkins to control Docker
    networks:
      - dev-net

  bitbucket:
    image: atlassian/bitbucket-server:7.6.0  # Replace with specific tag if needed
    container_name: bitbucket
    restart: always
    ports:
      - "7990:7990"
      - "7999:7999"
    volumes:
      - bitbucket_data:/var/atlassian/application-data/bitbucket
    networks:
      - dev-net
    environment:
      - BITBUCKET_HOME=/var/atlassian/application-data/bitbucket
      - JVM_SUPPORT_RECOMMENDED_ARGS=-Djava.security.egd=file:/dev/urandom

  dind:
    image: docker:19.03-dind  
    container_name: dind
    privileged: true  # Needed for DinD (Docker-in-Docker)
    networks:
      - dev-net
    environment:
      - DOCKER_TLS_CERTDIR=/certs
    volumes:
      - dind_data:/var/lib/docker

  webhook-listener:
    build: ./webhook  # You will run your Python webhook here
    container_name: webhook-listener
    volumes:
      - ./webhook:/app  # Mount your Python webhook code
      - ./logs:/app/logs  # Mount your Python webhook code
    networks:
      - dev-net
    ports:
      - "5000:5000"
    depends_on:
      - bitbucket
      - jenkins
    environment:
      FLASK_APP: app.py
      FLASK_ENV: development  # Enable development mode

networks:
  dev-net:
    driver: bridge

volumes:
  jenkins_home:
  bitbucket_data:
  dind_data:

Webhook Listener Setup webhook-listener/app.py contains the core logic for the webhook listener, it's idea is to get the REST API calls and detect for trigger commands (if present) in comments.

import logging
from flask import Flask, request, jsonify
import requests
from requests.auth import HTTPBasicAuth
import urllib.parse

# Configure logging to output to a file as well
logging.basicConfig(
    level=logging.DEBUG,  # Changed to DEBUG for more verbose logging
    format='%(asctime)s - %(levelname)s - %(message)s',
    handlers=[
        logging.FileHandler("webhook_listener.log"),  # Log to a file
        logging.StreamHandler()  # Also log to the console
    ]
)

app = Flask(__name__)

JENKINS_BASE_URL = "http://jenkins:8080"
JENKINS_USER = "YOUR USERNAME"
JENKINS_TOKEN = "YOUR JENKINS TOKEN"

REPO_TO_JOB_MAPPING = {
    'test1': 'TEST-JOB-CI/job/Test1',
    'test2': 'TEST-JOB-CI/job/Test2',
}

jenkins_auth = HTTPBasicAuth(JENKINS_USER, JENKINS_TOKEN)

def trigger_jenkins_pipeline(repo_name, branch_name):
    """Triggers a Jenkins build for the specified branch with parameters."""
    job_name = REPO_TO_JOB_MAPPING.get(repo_name)
    if not job_name:
        logging.error(f"No Jenkins job configured for repository: {repo_name}")
        return
    # Encode the branch name to handle special characters like '/'
    encoded_branch_name = urllib.parse.quote(branch_name, safe='')
    url = f"{JENKINS_BASE_URL}/job/{job_name}/job/{encoded_branch_name}/build"
    logging.debug(f"Triggering Jenkins build at URL: {url}")

    # Fetch Jenkins crumb for CSRF protection
    crumb_url = "http://jenkins:8080/crumbIssuer/api/xml?xpath=concat(//crumbRequestField,\":\",//crumb)"
    crumb_response = requests.get(crumb_url, auth=jenkins_auth)

    if crumb_response.status_code == 200:
        crumb_field, crumb_value = crumb_response.text.split(':')
        headers = {crumb_field: crumb_value}

        try:
            response = requests.post(url, auth=jenkins_auth, headers=headers)
            logging.debug(f"Response Code: {response.status_code}, Response Body: {response.text}")

            if response.status_code == 201:
                logging.info(f"Jenkins build triggered successfully for branch: {branch_name}")
            else:
                logging.error(f"Failed to trigger Jenkins build for branch: {branch_name}, Status Code: {response.status_code}, Reason: {response.reason}")
        except requests.exceptions.RequestException as e:
            logging.error(f"Request failed: {e}")
    else:
        logging.error(f"Failed to get Jenkins crumb, Status Code: {crumb_response.status_code}, Reason: {crumb_response.reason}")

@app.route('/bitbucket-webhook', methods=['POST'])
def bitbucket_webhook():
    """Handles incoming Bitbucket webhooks."""
    data = request.json
    logging.info("Webhook received from Bitbucket")
    logging.debug(f"Received data: {data}")

    if 'pullRequest' in data:
        pr = data['pullRequest']
        repo_name = pr['fromRef']['repository']['slug']
        branch_name = pr['fromRef']['displayId']
        comment = data.get('comment', {})

        logging.info(f"PR from repo: {repo_name}, branch: {branch_name} received")

        if comment:  # Check if there is a comment
            logging.info(f"Comment ID: {comment['id']}, Text: {comment['text']}, Author: {comment['author']['displayName']}")
            if "trigger" in comment['text'].lower():
                logging.info(f"Trigger word found in comment, triggering Jenkins build for repo: {repo_name}, branch: {branch_name}")
                trigger_jenkins_pipeline(repo_name, branch_name)
            else:
                logging.info("No trigger word found in comment")
        else:
            logging.info("No comment found in pull request")
    else:
        logging.warning("Request does not contain 'pullRequest' key")

    return jsonify({"status": "received"}), 200

if __name__ == '__main__':
    logging.info("Starting Bitbucket Webhook Listener...")
    app.run(host='0.0.0.0', port=5000)

Key Points:

It listens for POST requests from Bitbucket's webhook system.
Based on the repository name and branch from the webhook, it triggers the corresponding Jenkins pipeline.
The REPO_TO_JOB_MAPPING dictionary maps repository names to Jenkins job names, allowing for easy extension and flexibility.

webhook-listener/requirements.txt:

Flask==2.0.2
requests==2.26.0

Running the Setup Once everything is configured, run the following command from the project-root directory to spin up the services:

docker-compose up --build -d

This command will:

Build the Docker images for the webhook listener.
Start Jenkins, Bitbucket, and the webhook listener in isolated containers in detach mode.
Configure Test Repositories in Bitbucket
Setup multiple test repositories for the purpose of verification, say test1 and test2 with sample Jenkinsfile in each, which is to be used for setting up multibranch pipeline later
Configure the Webhooks
Setup a webhook for the endpoint http://<ip addr>:5000/bitbucket-webhook where ip address for the bitbcuket container can be retrieved by executing the following command

docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' bitbucket

Configure the webhook with following options

Jenkins Multibranch Pipeline Setup Once Jenkins is running on http://localhost:8080, you'll need to create multibranch pipelines for the repositories (test1, test2, etc.). The Jenkins job name will match the job names specified in the repo_to_job_mapping dictionary.
Go to Jenkins > New Item > Multibranch Pipeline
Configure each multibranch pipeline to scan the appropriate Bitbucket repository.
Setup branch source(here bitbucket which is already added through Manage Jenkins). Further setup the following to supress automatic build trigger based on new commits.

Test the Integration To test the webhook flow:
Create a repository in Bitbucket named test1.
Open a pull request for a branch (e.g., feature-branch) and provide the comment.
The webhook listener will detect the event and trigger the appropriate Jenkins job for the branch.
Monitor Jenkins to see the triggered job for that branch.

A sample docker webhook-listener log

$ docker-compose logs -f webhook-listener
webhook-listener  | 2024-09-22 10:17:09,128 - INFO - 172.24.0.3 - - [22/Sep/2024 10:17:09] "POST /bitbucket-webhook HTTP/1.1" 200 -
webhook-listener  | 2024-09-22 10:26:46,759 - INFO - Webhook received from Bitbucket
webhook-listener  | 2024-09-22 10:26:46,759 - DEBUG - Received data: {'eventKey': 'pr:comment:added', 'date': '2024-09-22T10:26:46+0000', 'actor': {'name': 'rmalhotra', 'emailAddress': 'rmalhotra@axiado.com', 'id': 1, 'displayName': 'r m', 'active': True, 'slug': 'rmalhotra', 'type': 'NORMAL', 'links': {'self': [{'href': 'http://localhost:7990/users/rmalhotra'}]}}, 'pullRequest': {'id': 2, 'version': 0, 'title': 'Feature/KWS-0002', 'description': '* Add repo details\r\n* KWS-0002: test commit', 'state': 'OPEN', 'open': True, 'closed': False, 'createdDate': 1727000643753, 'updatedDate': 1727000643753, 'fromRef': {'id': 'refs/heads/feature/KWS-0002', 'displayId': 'feature/KWS-0002', 'latestCommit': '89e876d3e6d0b84c96876a8e364705a30cd78771', 'repository': {'slug': 'test1', 'id': 2, 'name': 'test1', 'hierarchyId': 'f7550b4c5c2d0f8afefd', 'scmId': 'git', 'state': 'AVAILABLE', 'statusMessage': 'Available', 'forkable': True, 'project': {'key': 'TEST', 'id': 2, 'name': 'test', 'public': False, 'type': 'NORMAL', 'links': {'self': [{'href': 'http://localhost:7990/projects/TEST'}]}}, 'public': True, 'links': {'clone': [{'href': 'http://localhost:7990/scm/test/test1.git', 'name': 'http'}, {'href': 'ssh://git@localhost:7999/test/test1.git', 'name': 'ssh'}], 'self': [{'href': 'http://localhost:7990/projects/TEST/repos/test1/browse'}]}}}, 'toRef': {'id': 'refs/heads/develop', 'displayId': 'develop', 'latestCommit': '7ea6d999d56df26f381be449426a323f3d963c4c', 'repository': {'slug': 'test1', 'id': 2, 'name': 'test1', 'hierarchyId': 'f7550b4c5c2d0f8afefd', 'scmId': 'git', 'state': 'AVAILABLE', 'statusMessage': 'Available', 'forkable': True, 'project': {'key': 'TEST', 'id': 2, 'name': 'test', 'public': False, 'type': 'NORMAL', 'links': {'self': [{'href': 'http://localhost:7990/projects/TEST'}]}}, 'public': True, 'links': {'clone': [{'href': 'http://localhost:7990/scm/test/test1.git', 'name': 'http'}, {'href': 'ssh://git@localhost:7999/test/test1.git', 'name': 'ssh'}], 'self': [{'href': 'http://localhost:7990/projects/TEST/repos/test1/browse'}]}}}, 'locked': False, 'author': {'user': {'name': 'rmalhotra', 'emailAddress': 'rmalhotra@axiado.com', 'id': 1, 'displayName': 'r m', 'active': True, 'slug': 'rmalhotra', 'type': 'NORMAL', 'links': {'self': [{'href': 'http://localhost:7990/users/rmalhotra'}]}}, 'role': 'AUTHOR', 'approved': False, 'status': 'UNAPPROVED'}, 'reviewers': [], 'participants': [], 'links': {'self': [{'href': 'http://localhost:7990/projects/TEST/repos/test1/pull-requests/2'}]}}, 'comment': {'properties': {'repositoryId': 2}, 'id': 31, 'version': 0, 'text': 'now trigger', 'author': {'name': 'rmalhotra', 'emailAddress': 'rmalhotra@axiado.com', 'id': 1, 'displayName': 'r m', 'active': True, 'slug': 'rmalhotra', 'type': 'NORMAL', 'links': {'self': [{'href': 'http://localhost:7990/users/rmalhotra'}]}}, 'createdDate': 1727000806737, 'updatedDate': 1727000806737, 'comments': [], 'tasks': [], 'severity': 'NORMAL', 'state': 'OPEN'}}
webhook-listener  | 2024-09-22 10:26:46,759 - INFO - PR from repo: test1, branch: feature/KWS-0002 received
webhook-listener  | 2024-09-22 10:26:46,759 - INFO - Comment ID: 31, Text: now trigger, Author: r m
webhook-listener  | 2024-09-22 10:26:46,759 - INFO - Trigger word found in comment, triggering Jenkins build for repo: test1, branch: feature/KWS-0002
webhook-listener  | 2024-09-22 10:26:46,760 - DEBUG - Triggering Jenkins build at URL: http://jenkins:8080/job/Axiado-Test-CI/job/Test1/job/feature%2FKWS-0002/build

DEV Community: Rahul Malhotra

Automating CI/CD Pipelines with Bitbucket Webhooks and Jenkins: A Dockerized Demo