Secret Scanning in CI pipelines using Gitleaks and Pre-commit Hook.

In today's development environment, maintaining the security of your code is as crucial as ensuring its functionality. One of the key aspects of security is managing and safeguarding your secrets, such as API keys, passwords, and tokens. Accidentally committing these secrets to your repository can lead to severe security breaches. Implementing secret scanning in your Continuous Integration (CI) pipelines is essential to mitigate this risk. This blog will guide you through setting up secret scanning in GitLab CI pipelines using Gitleaks and a pre-commit hook.

Why Secret Scanning?

Before diving into the technical setup, let's briefly discuss why secret scanning is important:

1. Prevent Data Leaks: Secrets embedded in the code can be easily exposed if not managed properly, leading to unauthorized access.

2. Compliance: Many organizations have compliance requirements that mandate the protection of sensitive data.

3. Early Detection: Scanning for secrets early in the CI pipeline helps catch issues before they make it to production.

4. Faster Time to Market through Accelerated Releases: Automated secret scanning ensures that security checks do not become bottlenecks in the release process, allowing for faster and more frequent releases.

Tools Used

• Gitleaks: An open-source tool that scans your codebase for secrets, providing a comprehensive way to detect and prevent leaks.

• Git Hooks: This is a git functionality. It's a way to run custom scripts when certain actions occur.

• OWASP Juice Shop: The application code to be used for this demonstration. It is probably the most modern and sophisticated insecure web application. Juice Shop encompasses vulnerabilities from the entire OWASP Top Ten and many other security flaws in real-world applications!

• Gitlab: A web-based DevOps lifecycle tool that provides a Git repository manager with features like source code management (SCM), continuous integration (CI), continuous deployment (CD), and more.

STEP 1:
Fork the OWASP Juice Shop Gitlab repository

Added .env file that contains some dummy secrets to the repo.

STEP 2:
Create a .gitlab-ci.yml file in the repository. This file is the configuration file that GitLab CI/CD uses to define the pipeline and its various stages, jobs, and actions.

variables:
    IMAGE_NAME: sirlawdin/juice-shop-app
    IMAGE_TAG: juice-shop-1.1

stages:
    - cache
    - test
    - build 

create_cache:
    image: node:18-bullseye
    stage: cache
    script:
      - yarn install
    cache:
      key:
        files:
          - yarn.lock
      paths:
        - node_modules/
        - yarn.lock
        - .yarn
      policy: pull-push

gitleaks:
  stage: test
  image:
    name: zricethezav/gitleaks
    entrypoint: [""]
  script: 
    - gitleaks detect --source . --verbose --report-path gitleaks-report.json

yarn_test:
    image: node:18-bullseye
    stage: test
    script: 
      - yarn install
      - yarn test
    cache:
      key:
        files:
          - yarn.lock
      paths:
        - node_modules/
        - yarn.lock
        - .yarn
      policy: pull-push

build_image:
    stage: build
    image: docker:24
    services:
      - docker:24-dind
    variables:
        DOCKER_USER: sirlawdin
    before_script:
      - echo $DOCKER_PASS | docker login -u $DOCKER_USER --password-stdin
    script:
      - docker build -t $IMAGE_NAME:$IMAGE_TAG .
      - docker push $IMAGE_NAME:$IMAGE_TAG

Here's a brief explanation of each section:

IMAGE_NAME and IMAGE_TAG: These variables define the name and tag of the Docker image that will be built and pushed.

variables:
    IMAGE_NAME: sirlawdin/juice-shop-app
    IMAGE_TAG: juice-shop-1.1

cache: Stage for caching dependencies.
test: Stage for running tests and performing secret scanning.
build: Stage for building and pushing the Docker image.

stages:
    - cache
    - test
    - build 

create_cache: This job uses the node:18-bullseye image to install dependencies with Yarn.
Caching: The node_modules/, yarn.lock, and .yarn directories are cached based on the yarn.lock file to speed up future pipeline runs.

create_cache:
    image: node:18-bullseye
    stage: cache
    script:
      - yarn install
    cache:
      key:
        files:
          - yarn.lock
      paths:
        - node_modules/
        - yarn.lock
        - .yarn
      policy: pull-push

gitleaks: This job uses the zricethezav/gitleaks image to scan the source code for secrets.
Report: The results are saved to gitleaks-report.json.

gitleaks:
  stage: test
  image:
    name: zricethezav/gitleaks
    entrypoint: [""]
  script: 
    - gitleaks detect --source . --verbose --report-path gitleaks-report.json

yarn_test: This job installs dependencies and runs tests using the node:18-bullseye image.
Caching: Similar to the create_cache job, it caches dependencies to speed up future runs.

yarn_test:
    image: node:18-bullseye
    stage: test
    script: 
      - yarn install
      - yarn test
    cache:
      key:
        files:
          - yarn.lock
      paths:
        - node_modules/
        - yarn.lock
        - .yarn
      policy: pull-push

build_image: This job uses the docker:24 image and the Docker-in-Docker (dind) service to build and push the Docker image.
Docker Login: The DOCKER_PASS and DOCKER_USER variables are used to log in to Docker Hub.
Build and Push: The Docker image is built with the specified IMAGE_NAME and IMAGE_TAG, and then pushed to the Docker registry.

build_image:
    stage: build
    image: docker:24
    services:
      - docker:24-dind
    variables:
        DOCKER_USER: sirlawdin
    before_script:
      - echo $DOCKER_PASS | docker login -u $DOCKER_USER --password-stdin
    script:
      - docker build -t $IMAGE_NAME:$IMAGE_TAG .
      - docker push $IMAGE_NAME:$IMAGE_TAG

After running the pipeline, the job at the gitleak job failed due to leaks found in the repository.

PS: The dummy secrets previously added to the repository were all flagged.

Fetching changes with git depth set to 20...
Initialized empty Git repository in /builds/Sirlawdin/juice-shop/.git/
Created fresh repository.
Checking out 5c0f71b8 as detached HEAD (ref is main)...
Skipping Git submodules setup
$ git remote set-url origin "${CI_REPOSITORY_URL}"
Executing "step_script" stage of the job script
00:08
Using docker image sha256:7da57b8e4dc2b857722c7fe447673bd938d452e61acbeb652bf90a210385457a for zricethezav/gitleaks with digest zricethezav/gitleaks@sha256:75bdb2b2f4db213cde0b8295f13a88d6b333091bbfbf3012a4e083d00d31caba ...
$ gitleaks detect --verbose --source .
    ○
    │╲
    │ ○
    ○ ░
    ░    gitleaks
Finding:     API_KEY=T8DWKJKAWUQ27MDSL902DBJAL
USERNAME= JaneDoe
Secret:      T8DWKJKAWUQ27MDSL902DBJAL
RuleID:      generic-api-key
Entropy:     3.973661
File:        .env
Line:        1
Commit:      5c0f71b87695579ad0d6999d485cd7b659ae6be6
Author:      Salaudeen O. Abdulrasaq
Email:       [e-mail]
Date:        2024-07-16T04:30:31Z
Fingerprint: 5c0f71b87695579ad0d6999d485cd7b659ae6be6:.env:generic-api-key:1
Finding:     SECRET_KEY=IFW2dSNA02JD7BDJ2BCJLA
Secret:      IFW2dSNA02JD7BDJ2BCJLA
RuleID:      generic-api-key
Entropy:     3.754442
File:        .env
Line:        4
Commit:      5c0f71b87695579ad0d6999d485cd7b659ae6be6
Author:      Salaudeen O. Abdulrasaq
Email:       [e-mail]
Date:        2024-07-16T04:30:31Z
Fingerprint: 5c0f71b87695579ad0d6999d485cd7b659ae6be6:.env:generic-api-key:4
Finding:     ...2dSNA02JD7BDJ2BCJLA
ACCESS_KEY=ROHXMPGWQLAHJI92NJSD
Secret:      ROHXMPGWQLAHJI92NJSD
RuleID:      generic-api-key
Entropy:     4.121928
File:        .env
Line:        5
Commit:      5c0f71b87695579ad0d6999d485cd7b659ae6be6
Author:      Salaudeen O. Abdulrasaq
Email:       [e-mail]
Date:        2024-07-16T04:30:31Z
Fingerprint: 5c0f71b87695579ad0d6999d485cd7b659ae6be6:.env:generic-api-key:5
2:09PM INF 33 commits scanned.
2:09PM INF scan completed in 7.61s
2:09PM WRN leaks found: 80
Cleaning up project directory and file based variables
00:00
ERROR: Job failed: exit code 1

PRE_COMMIT HOOK
Implementing a pre-commit hook to scan for potential leaks in your code can prevent sensitive credentials from being committed to the repository. This proactive measure ensures sensitive information remains secure and never makes it to the repository commits, mitigating the risk of exposure to attackers.

Git Hooks; There are various types of hooks available in Git, including:

• Pre-commit
• Pre-push
• Pre-rebase
• And more...

Pre-commit hook is fired when changes made to a repo is about to be committed.

To create a pre-commit hook:

• Open your terminal and navigate to the root directory of your Git repository.
  cd /path/to/your/repository

• Create the pre-commit hook file:
  Inside the .git/hooks directory, create a file named pre-commit.
  vi .git/hooks/pre-commit

Enter the below script into the pre-commit file

#!/bin/sh

# Configuration
GITLEAKS_IMAGE="zricethezav/gitleaks"
HOST_FOLDER_TO_SCAN="juice-shop-repo"
CONTAINER_FOLDER="/path"
REPORT_FILE="gitleaks-report.json"

# Function to display an error message and exit
exit_with_error() {
    echo "$1"
    exit 1
}

# Run Gitleaks
echo "Running Gitleaks for secret scanning..."
docker pull ${GITLEAKS_IMAGE} || exit_with_error "Failed to pull Gitleaks Docker image."

# Run Gitleaks container
docker run -v ${HOST_FOLDER_TO_SCAN}:${CONTAINER_FOLDER} ${GITLEAKS_IMAGE} detect --source="${CONTAINER_FOLDER}" --verbose --report-path=${REPORT_FILE}
GITLEAKS_EXIT_CODE=$?

# Check if Gitleaks detected any secrets
if [ ${GITLEAKS_EXIT_CODE} -ne 0 ]; then
    echo "Gitleaks detected secrets in your code. Please fix the issues before committing."
    cat ${REPORT_FILE}
    exit 1
fi

echo "Gitleaks found no secrets. Proceeding with commit."
exit 0

Explanation:

Configuration:
Defined variables for image name, host folder, container folder, and report file for easy configuration and readability.

Error Handling:
The function exit_with_error was added to handle errors gracefully and provide meaningful error messages.

Docker Pull:
Added error handling for the docker pull command to ensure the script exits if pulling the Gitleaks image fails.

Docker Run:
It is used to configure variables to run the Gitleaks container and capture its exit code.

Check Gitleaks Exit Code:
Checked the exit code of the Gitleaks run and displayed the report if secrets were detected, preventing the commit.

• Make the hook executable; Change the file permissions to make the pre-commit hook executable.

chmod +x .git/hooks/pre-commit

The pre-commit hook has helped enforce secret protection by preventing commits until any leaked secrets are removed.

Now, your pre-commit hook will run every time you attempt to commit changes, helping to enforce code quality and security checks before the commit is finalized.