Automating Breaking Change Detection from Git History: A Lightweight, CI-First Approach

In modern software development, unexpected dependency drift coupled with source code modifications remains a frequent source of regressions. Pull requests that alter core library versions while simultaneously modifying consumer code often slip through review without adequate scrutiny, causing deployment failures or subtle runtime defects.

To mitigate this without introducing heavyweight infrastructure, we engineered a standalone, stateless CLI tool designed to inspect Git revision trees and detect breaking changes between commits within seconds.

Below is an in-depth breakdown of the design requirements, the edge cases encountered during development, the complete implementation using Python and Pydantic v2, and its integration into CI workflows.

---

1. Architectural & Design Constraints

When architecting this utility, four core constraints were established:

1. Zero Persistent Processes: No background daemons, sidecars, or state stores. The tool must operate as a strictly stateless CLI command.
2. Predictable Execution Speed: Git trees must be queried directly via low-level primitives, scanning large diffs in under 10 seconds.
3. Execution Security: Strict prevention of path injection and directory traversal attacks when parsing path arguments.
4. CI-First Design: Diagnostics must be delivered in machine-readable JSON format, enabling downstream jobs (e.g., automated PR comments, bot alerts, pipeline blockers) to consume outputs deterministically.

flowchart TD
    subgraph InputStage ["1. Input & Sanitization"]
        CLI["CLI Arguments: Target Dir, Commit Range, Package Path"]
        Sanitize["Path Traversal Defense: resolve() & is_relative_to()"]
        CLI -- "Pass arguments" --> Sanitize
    end

    subgraph GitStage ["2. Git Tree Inspection"]
        DiffTree["git diff-tree --name-only -r"]
        FallbackDiff["Fallback: git diff --name-only HEAD"]
        GitShow["git show target_commit:package.json"]

        Sanitize -- "Validated paths" --> DiffTree
        DiffTree -. "Failed / Shallow clone" .-> FallbackDiff
        DiffTree -- "Changed files detected" --> GitShow
        FallbackDiff -- "Working tree diff" --> GitShow
    end

    subgraph AnalysisStage ["3. Dependency Analysis & Output"]
        ParseJSON["Fast JSON Deserialization: orjson / stdlib fallback"]
        DiffCompute["Compare dependencies & devDependencies"]
        PydanticModel["Pydantic v2 Validation: RiskAlertOutput"]
        ExitCode["Process Exit: 0 (Clean) / 1 (Breaking Change)"]

        GitShow -- "Raw file contents" --> ParseJSON
        ParseJSON -- "Parsed dicts" --> DiffCompute
        DiffCompute -- "Structured diffs" --> PydanticModel
        PydanticModel -- "Serialized JSON stdout" --> ExitCode
    end

---

2. Real-World Edge Cases and Debugging Challenges

Building tools around Git primitives often reveals discrepancies between local developer environments and remote execution contexts. Several non-obvious challenges surfaced during implementation.

Edge Case 1: Commit Range Inconsistencies and Shallow Clones

Standard diff range specifications (such as HEAD~1..HEAD) exhibit divergent behavior depending on branch topology, rebase operations, and detached HEAD states.

More critically, in Continuous Integration environments (e.g., GitHub Actions using default actions/checkout), repositories are fetched with fetch-depth: 1. In a shallow clone, parent commits do not exist in the object database, causing commands like git show HEAD~1:package.json to fail immediately.

Solution:
We adopted a multi-stage fallback strategy. The script first attempts git diff-tree --no-commit-id --name-only -r <commit_range>. If this returns an error or empty output, it gracefully degrades to checking the local working tree via git diff --name-only HEAD. Furthermore, when retrieving historical versions of package.json, if parent commit extraction fails, the script falls back to reading the existing file from the filesystem.

Edge Case 2: Serialization Latency vs. Portability (orjson Dynamic Fallback)

In large repositories with extensive dependency lists and hundreds of altered files, repeated serialization of diagnostic outputs can introduce noticeable overhead when relying on Python's built-in json module. While orjson offers substantial throughput improvements due to its Rust-backed implementation, requiring it as a mandatory dependency would complicate execution across minimal Docker images and ad-hoc runner environments.

Solution:
We implemented resilient wrapper functions (safe_dumps and safe_loads) with dynamic runtime fallback:

try:
    import orjson
    HAS_ORJSON = True
except ImportError:
    import json
    HAS_ORJSON = False

If orjson is available, high-speed binary serialization is utilized; otherwise, the standard library handles the payload without interruption.

Edge Case 3: Directory Traversal and Path Injection Vulnerabilities

Because CLI arguments allow users and CI scripts to specify arbitrary target directories and paths to configuration files (such as package.json), accepting untrusted paths introduces security risks (e.g., malicious paths like ../../etc/passwd or symbolic link escapes).

Solution:
Paths are normalized using pathlib.Path.resolve(), which resolves all symbolic links and relative path tokens (..). The path is subsequently evaluated against the current working directory (cwd) using is_relative_to():

resolved_path = (cwd / pkg_path_str).resolve()
if not resolved_path.is_relative_to(cwd):
    raise ValueError("Path traversal detected.")

If validation fails, the script safely falls back to standard repository defaults rather than executing unauthorized file lookups.

---

3. Complete Implementation

The following single-file script incorporates all design patterns, defensive fallbacks, and Pydantic v2 schemas:

import sys
import subprocess
from pathlib import Path
from typing import List, Dict, Set, Optional

# ==========================================
# Dynamic Import with orjson Fallback
# ==========================================
try:
    import orjson
    HAS_ORJSON = True
except ImportError:
    import json
    HAS_ORJSON = False

from pydantic import BaseModel, Field


# ==========================================
# Pydantic v2 Schema Definitions
# ==========================================

class DependencyDiff(BaseModel):
    package_name: str
    previous_version: Optional[str] = None
    current_version: Optional[str] = None
    change_type: str = Field(..., description="added, removed, or updated")


class ScanTargetConfig(BaseModel):
    target_directory: str = Field(default="src/", description="Target directory path to inspect")
    package_json_path: str = Field(default="package.json", description="Path to package.json")
    commit_range: str = Field(default="HEAD~1..HEAD", description="Git commit range for inspection")


class RiskAlertOutput(BaseModel):
    has_breaking_change: bool = Field(..., description="Flag indicating detected breaking changes")
    target_directory: str
    changed_files: List[str]
    dependency_diffs: List[DependencyDiff]
    warning_message: Optional[str] = None


# ==========================================
# Core Utility Functions
# ==========================================

def safe_dumps(data: dict) -> bytes:
    """Serialize dictionary to bytes using orjson if available, otherwise stdlib json."""
    if HAS_ORJSON:
        return orjson.dumps(data, option=orjson.OPT_INDENT_2)
    else:
        return json.dumps(data, ensure_ascii=False, indent=2).encode("utf-8")


def safe_loads(content: bytes | str) -> dict:
    """Deserialize JSON payload to dictionary with graceful error handling."""
    if not content:
        return {}
    try:
        if isinstance(content, bytes):
            content = content.decode("utf-8")

        if HAS_ORJSON:
            return orjson.loads(content)
        else:
            return json.loads(content)
    except Exception:
        return {}


def run_git_command(args: List[str]) -> Optional[str]:
    """Execute Git subcommands safely with shell=False to prevent command injection."""
    try:
        result = subprocess.run(
            ["git"] + args,
            stdout=subprocess.PIPE,
            stderr=subprocess.PIPE,
            text=True,
            check=True,
            shell=False
        )
        return result.stdout.strip()
    except (subprocess.CalledProcessError, FileNotFoundError):
        return None


def get_changed_files(commit_range: str) -> List[str]:
    """Retrieve list of modified files within the specified commit range."""
    output = run_git_command(["diff-tree", "--no-commit-id", "--name-only", "-r", commit_range])
    if not output:
        output_status = run_git_command(["diff", "--name-only", "HEAD"])
        if not output_status:
            return []
        return output_status.splitlines()
    return output.splitlines()


def get_file_content_at_commit(commit_ref: str, file_path: Path) -> Optional[Dict]:
    """Extract and parse file contents at a specific Git revision."""
    git_file_path = file_path.as_posix()
    content = run_git_command(["show", f"{commit_ref}:{git_file_path}"])
    if content is None:
        return None
    return safe_loads(content)


def analyze_dependencies(package_json_path: Path, commit_range: str) -> List[DependencyDiff]:
    """Calculate the delta between dependency states across commit boundaries."""
    commits = commit_range.split("..")
    base_commit = commits[0] if len(commits) > 0 and commits[0] else "HEAD~1"
    target_commit = commits[1] if len(commits) > 1 and commits[1] else "HEAD"

    current_data = get_file_content_at_commit(target_commit, package_json_path)
    base_data = get_file_content_at_commit(base_commit, package_json_path)

    # Fallback to local working directory if commit inspection yields no data
    if not current_data and package_json_path.exists():
        try:
            with open(package_json_path, "rb") as f:
                current_data = safe_loads(f.read())
        except Exception:
            current_data = {}

    if not current_data:
        current_data = {}

    current_deps = {**current_data.get("dependencies", {}), **current_data.get("devDependencies", {})}
    base_deps = {}
    if base_data:
        base_deps = {**base_data.get("dependencies", {}), **base_data.get("devDependencies", {})}

    diffs: List[DependencyDiff] = []
    all_keys: Set[str] = set(current_deps.keys()) | set(base_deps.keys())

    for pkg in sorted(all_keys):
        prev_v = base_deps.get(pkg)
        curr_v = current_deps.get(pkg)

        if prev_v is None and curr_v is not None:
            diffs.append(DependencyDiff(package_name=pkg, current_version=curr_v, change_type="added"))
        elif prev_v is not None and curr_v is None:
            diffs.append(DependencyDiff(package_name=pkg, previous_version=prev_v, change_type="removed"))
        elif prev_v != curr_v:
            diffs.append(DependencyDiff(package_name=pkg, previous_version=prev_v, current_version=curr_v, change_type="updated"))

    return diffs


def main():
    target_dir = sys.argv[1] if len(sys.argv) > 1 else "src/"
    commit_range = sys.argv[2] if len(sys.argv) > 2 else "HEAD~1..HEAD"
    pkg_path_str = sys.argv[3] if len(sys.argv) > 3 else "package.json"

    # ==========================================
    # Path Traversal Guard
    # ==========================================
    cwd = Path.cwd().resolve()

    try:
        resolved_path = (cwd / pkg_path_str).resolve()
        if not resolved_path.is_relative_to(cwd):
            raise ValueError("Path traversal detected.")
        package_json_path = resolved_path
    except (ValueError, RuntimeError):
        package_json_path = cwd / "package.json"

    config = ScanTargetConfig(
        target_directory=target_dir,
        package_json_path=str(package_json_path),
        commit_range=commit_range
    )

    # 1. Identify changed paths
    changed_files = get_changed_files(config.commit_range)

    # 2. Filter paths matching the target directory
    target_prefix = config.target_directory.lstrip("./")
    affected_files = [f for f in changed_files if f.startswith(target_prefix)]
    has_target_changes = len(affected_files) > 0

    # 3. Analyze dependency variations
    dependency_diffs = []
    if has_target_changes:
        dependency_diffs = analyze_dependencies(package_json_path, config.commit_range)

    # 4. Determine breaking change condition
    has_breaking_change = has_target_changes and len(dependency_diffs) > 0

    warning_msg = None
    if has_breaking_change:
        warning_msg = (
            f"CRITICAL: Simultaneous modifications detected in '{config.target_directory}' and dependencies. "
            "Verify all test suites and consumers prior to merging."
        )

    # 5. Populate and validate output model
    output_model = RiskAlertOutput(
        has_breaking_change=has_breaking_change,
        target_directory=config.target_directory,
        changed_files=affected_files,
        dependency_diffs=dependency_diffs,
        warning_message=warning_msg
    )

    # 6. Structured serialization to stdout
    output_dict = output_model.model_dump()
    json_bytes = safe_dumps(output_dict)
    sys.stdout.buffer.write(json_bytes)
    sys.stdout.buffer.write(b"\n")

    # Exit code signaling for CI pipelines
    if has_breaking_change:
        sys.exit(1)
    else:
        sys.exit(0)


if __name__ == "__main__":
    main()

💡 For immediate deployment: The complete source code suite (ZIP) for this architecture is available on Gumroad for $0+ (Pay What You Want).

---

4. CI/CD Integration

To leverage this check as an automated gate in GitHub Actions, configure the workflow to fetch the complete commit history (fetch-depth: 0) so that range diffs can be evaluated accurately against the base branch.

name: Breaking Change Gate

on:
  pull_request:
    branches: [ main ]

jobs:
  check-breaking-changes:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout repository
        uses: actions/checkout@v4
        with:
          fetch-depth: 0 # Retrieve full history to ensure base commit availability

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.11'

      - name: Install dependencies
        run: pip install pydantic orjson

      - name: Run Breaking Change Detector
        run: python detector.py "src/" "origin/main...HEAD" "package.json"

Sample Output

When modifications in src/ coincide with dependency version changes in package.json, the detector emits the following structured JSON and exits with return code 1:

{
  "has_breaking_change": true,
  "target_directory": "src/",
  "changed_files": [
    "src/components/Button.tsx"
  ],
  "dependency_diffs": [
    {
      "package_name": "lodash",
      "previous_version": "^4.17.21",
      "current_version": "^5.0.0",
      "change_type": "updated"
    }
  ],
  "warning_message": "CRITICAL: Simultaneous modifications detected in 'src/' and dependencies. Verify all test suites and consumers prior to merging."
}

This structured output can be piped directly into utilities such as jq, logged to monitoring endpoints, or used by subsequent pipeline steps to format targeted pull request comments.

---

If this engineering log saved your production server (and your sanity), consider supporting our architecture on GitHub Sponsors.