Building a Custom Git Pre-Commit Secret Scanner: Entropy and Regex Analysis

When managing source code, a 0.1-second lapse in concentration can lead to catastrophic infrastructure compromises. While various commercial solutions exist, building a lightweight, custom secret scanner allows you to deeply integrate security checks into your specific workflow without external dependencies.

Below is a complete implementation of a Git diff scanner that leverages both pattern matching (Regex) and Shannon entropy calculations to detect hardcoded credentials before they ever leave your local machine.

The Core Implementation

This Python script inspects the current Git diff, matching it against known credential patterns and calculating the entropy of string literals to catch unknown, high-entropy tokens.

import re
import math
import subprocess
import sys
import json

# WAF/DLP safe pattern definitions using string concatenation
PATTERNS = [
    {
        "name": "AWS Access Key", 
        "pattern": re.compile(r'AK' + r'IA[0-9A-Z]{16}'), 
        "severity": "CRITICAL"
    },
    {
        "name": "Slack Token", 
        "pattern": re.compile(r'xox[baprs]-' + r'[0-9a-zA-Z]{10,48}'), 
        "severity": "CRITICAL"
    },
    {
        "name": "Generic API Key / Token", 
        "pattern": re.compile(r'(?i)(?:api[_-]?key|access[_-]?token|auth[_-]?token|bearer)\s*[:=]\s*["\']([a-zA-Z0-9_\-\.\/]{16,64})["\']'), 
        "severity": "HIGH"
    },
    {
        "name": "Password in Assignment", 
        "pattern": re.compile(r'(?i)(?:password|passwd|pwd)\s*[:=]\s*["\']([^"\']{8,})["\']'), 
        "severity": "HIGH"
    }
]

def calculate_shannon_entropy(data):
    """
    Calculates the Shannon entropy of a given string.
    High entropy indicates a high degree of randomness, typical of cryptographic keys.
    """
    if not data:
        return 0
    entropy = 0
    length = len(data)
    for x in set(data):
        p_x = float(data.count(x)) / length
        entropy += - p_x * math.log2(p_x)
    return entropy

def get_git_diff():
    """
    Extracts the current git diff. 
    Attempts HEAD, cached (staged), and unstaged diffs in sequence.
    """
    commands = [
        ["git", "diff", "HEAD"], 
        ["git", "diff", "--cached"], 
        ["git", "diff"]
    ]
    for cmd in commands:
        try:
            result = subprocess.run(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True, check=True, shell=False)
            if result.stdout.strip():
                return result.stdout
        except (subprocess.CalledProcessError, FileNotFoundError):
            continue
    return ""

def scan_diff(diff_output):
    """
    Scans the diff output line by line against regex patterns and entropy thresholds.
    """
    findings = []
    lines = diff_output.splitlines()
    current_file = "unknown"

    for line in lines:
        # Track the file currently being scanned
        if line.startswith("+++ b/"):
            current_file = line[6:]
            continue

        # Only scan added lines; ignore context and deletions
        if line.startswith("+") and not line.startswith("++") and not line.startswith("+---"):
            content = line[1:]

            # 1. Regex Pattern Matching
            for p in PATTERNS:
                match = p["pattern"].search(content)
                if match:
                    findings.append({
                        "file": current_file, 
                        "type": p["name"], 
                        "severity": p["severity"], 
                        "matched": match.group(0)[:10] + "..."
                    })

            # 2. Shannon Entropy Analysis
            # Extract potential tokens enclosed in quotes
            tokens = re.findall(r'["\']([a-zA-Z0-9_\-\.\/\+=]{16,})["\']', content)
            for token in tokens:
                if calculate_shannon_entropy(token) > 4.5:
                    findings.append({
                        "file": current_file, 
                        "type": "High Entropy String", 
                        "severity": "MEDIUM", 
                        "matched": token[:6] + "..."
                    })

    return findings

def main():
    diff_data = get_git_diff()
    if not diff_data:
        sys.exit(0)

    vulnerabilities = scan_diff(diff_data)

    if vulnerabilities:
        print(json.dumps({
            "vulnerabilities_found": len(vulnerabilities), 
            "findings": vulnerabilities
        }, indent=2))

    # Block the commit/pipeline if HIGH or CRITICAL severity is detected
    if any(f["severity"] in ["HIGH", "CRITICAL"] for f in vulnerabilities):
        sys.exit(1)

if __name__ == "__main__":
    main()

Architectural Workflow and Integration Strategy

To visualize how this defensive mechanism integrates into the development lifecycle, we can map out the interception points. The scanner operates as a dual-layer defense: primarily as a local blocker, and secondarily as a CI/CD safeguard.

flowchart TD
    subgraph LocalEnv ["Local Development Environment"]
        A["Developer executes 'git commit'"]
        B["pre-commit Hook Triggered"]
        C["Git Diff Extraction"]
        D["Regex & Entropy Scanning"]
    end

    subgraph CIEnv ["CI/CD Pipeline (Fallback)"]
        E["Pull Request Created"]
        F["Automated Secret Scan"]
    end

    A -- "Initiates commit process" --> B
    B -- "Extracts staged changes" --> C
    C -- "Passes diff data" --> D

    D -- "Secrets detected" --> G["Abort Commit (exit 1)"]
    D -- "Clean diff" --> H["Commit Successful"]

    H -. "Push to remote repository" .-> E
    E -- "Triggers pipeline" --> F
    F -- "Secrets detected" --> I["Fail Build (exit 1)"]
    F -- "Clean PR" --> J["Allow Merge"]

Operational Guidelines

The primary objective of implementing this tooling is not to establish a punitive system for developers. Rather, it is to systematically engineer a safety net—providing the psychological security that even if a human error occurs, the automated pipeline will inevitably halt the breach before it reaches a remote repository.

The most effective deployment strategy is embedding this script directly into your Git pre-commit hooks, ensuring it executes synchronously precisely at the moment of a commit.

For continuous integration environments (CI/CD), configure the workflow to invoke this script upon Pull Request creation. If the script returns an exit 1 due to high-severity findings, the pipeline should automatically fail, blocking the merge operation until the offending commits are rewritten or expunged from the Git history.

There is no such thing as a perfect defense, and entropy analysis can occasionally yield false positives on complex hashes. However, mitigating the catastrophic risk of a compromised repository with a 10-second automation investment is an essential architectural decision for any mature engineering team.

---

If this engineering log saved your production server (and your sanity), consider supporting our architecture on GitHub Sponsors.