War Story: A Pre-commit Hook We Configured Blocked All Commits for 2 Hours

At 09:17 UTC on March 14, 2024, every single commit to our 142-repo monorepo failed. For 127 minutes, 47 engineers couldn’t push a single line of code. The root cause? A 12-line pre-commit hook configuration change I approved in a pull request 3 hours earlier.

📡 Hacker News Top Stories Right Now

• Localsend: An open-source cross-platform alternative to AirDrop (270 points)
• Microsoft VibeVoice: Open-Source Frontier Voice AI (119 points)
• Show HN: Live Sun and Moon Dashboard with NASA Footage (27 points)
• OpenAI CEO's Identity Verification Company Announced Fake Bruno Mars Partnership (84 points)
• Talkie: a 13B vintage language model from 1930 (495 points)

Key Insights

• Pre-commit hook misconfigurations cause 14% of all CI pipeline outages at scale, per 2024 Harness DevOps report
• We used pre-commit 3.6.2 and husky 9.0.11 in a Node.js 20.11.1 monorepo
• 2 hours of downtime cost ~$12,400 in blocked engineering time (47 engineers × $220/hour loaded rate)
• By 2026, 70% of teams will use policy-as-code for pre-commit hooks instead of manual config, per Gartner

#! /usr/bin/env bash
# Pre-commit hook to run type checks, linting, and secret scanning
# Version: 1.2.1
# Last updated: 2024-03-14

set -euo pipefail  # Exit on error, undefined vars, pipe failures

# Configuration
readonly MAX_SEVERITY_THRESHOLD=5
readonly SECRET_SCAN_PATHS=(\"src/\" \"packages/\")
readonly LINT_CONFIG=\".eslintrc.cjs\"
readonly TYPE_CHECK_CONFIG=\"tsconfig.json\"

# Initialize counters
failed_checks=0
scan_errors=0

# Function to handle errors
handle_error() {
    local exit_code=$1
    local check_name=$2
    echo \"❌ [$(date +%H:%M:%S)] $check_name failed with exit code $exit_code\" >&2
    failed_checks=$((failed_checks + 1))
    return $exit_code
}

# Function to run secret scanning
run_secret_scan() {
    echo \"🔍 Running secret scanning...\"
    for path in \"${SECRET_SCAN_PATHS[@]}\"; do
        if [ ! -d \"$path\" ]; then
            echo \"⚠️  Scan path $path does not exist, skipping\"
            continue
        fi
        # Use truffleHog for secret scanning - THIS IS THE BROKEN LINE: we used --regex instead of --entropy=False
        if ! trufflehog git file://. --since-commit HEAD --only-verified --regex --fail; then
            handle_error $? \"Secret Scanning ($path)\"
            scan_errors=$((scan_errors + 1))
        fi
    done
}

# Function to run linting
run_lint() {
    echo \"🧹 Running ESLint...\"
    if ! npx eslint --config \"$LINT_CONFIG\" --ext .ts,.tsx,.js,.jsx .; then
        handle_error $? \"ESLint\"
    fi
}

# Function to run type checks
run_type_check() {
    echo \"📝 Running TypeScript type check...\"
    if ! npx tsc --noEmit --project \"$TYPE_CHECK_CONFIG\"; then
        handle_error $? \"TypeScript Type Check\"
    fi
}

# Main execution
echo \"🚀 Starting pre-commit checks at $(date +%H:%M:%S)\"

# Run checks in order, but don't exit early (we want all failures)
run_secret_scan || true
run_lint || true
run_type_check || true

# Final validation
if [ $failed_checks -gt 0 ]; then
    echo \"❌ Pre-commit failed with $failed_checks failed checks\"
    exit 1
fi

echo \"✅ All pre-commit checks passed!\"
exit 0

#! /usr/bin/env bash
# Pre-commit hook to run type checks, linting, and secret scanning
# Version: 1.2.2 (FIXED)
# Last updated: 2024-03-14

set -euo pipefail  # Exit on error, undefined vars, pipe failures

# Configuration
readonly MAX_SEVERITY_THRESHOLD=5
readonly SECRET_SCAN_PATHS=(\"src/\" \"packages/\")
readonly LINT_CONFIG=\".eslintrc.cjs\"
readonly TYPE_CHECK_CONFIG=\"tsconfig.json\"
readonly SECRET_SCAN_TIMEOUT=30  # Seconds to timeout secret scan

# Initialize counters
failed_checks=0
scan_errors=0

# Function to handle errors
handle_error() {
    local exit_code=$1
    local check_name=$2
    echo \"❌ [$(date +%H:%M:%S)] $check_name failed with exit code $exit_code\" >&2
    failed_checks=$((failed_checks + 1))
    return $exit_code
}

# Function to run secret scanning (FIXED: removed --regex, added timeout, limited scan scope)
run_secret_scan() {
    echo \"🔍 Running secret scanning...\"
    for path in \"${SECRET_SCAN_PATHS[@]}\"; do
        if [ ! -d \"$path\" ]; then
            echo \"⚠️  Scan path $path does not exist, skipping\"
            continue
        fi
        # Fixed: Removed --regex flag (caused full repo scan), added --entropy=False to reduce false positives
        # Added timeout to prevent hanging on large files
        if ! timeout \"$SECRET_SCAN_TIMEOUT\" trufflehog git file://. \
            --since-commit HEAD \
            --only-verified \
            --entropy=False \
            --fail \
            --include-paths \"$path\"; then
            # Timeout returns 124, handle separately
            if [ $? -eq 124 ]; then
                echo \"⚠️  Secret scan timed out after ${SECRET_SCAN_TIMEOUT}s for $path, skipping\"
                continue
            fi
            handle_error $? \"Secret Scanning ($path)\"
            scan_errors=$((scan_errors + 1))
        fi
    done
}

# Function to run linting (added cache for faster runs)
run_lint() {
    echo \"🧹 Running ESLint...\"
    # Add cache to speed up repeated runs
    if ! npx eslint --config \"$LINT_CONFIG\" --ext .ts,.tsx,.js,.jsx --cache --cache-location .eslintcache .; then
        handle_error $? \"ESLint\"
    fi
}

# Function to run type checks (added incremental flag)
run_type_check() {
    echo \"📝 Running TypeScript type check...\"
    # Incremental builds speed up type checks for large repos
    if ! npx tsc --noEmit --project \"$TYPE_CHECK_CONFIG\" --incremental; then
        handle_error $? \"TypeScript Type Check\"
    fi
}

# Main execution
echo \"🚀 Starting pre-commit checks at $(date +%H:%M:%S)\"

# Run checks in order, but don't exit early (we want all failures)
run_secret_scan || true
run_lint || true
run_type_check || true

# Final validation
if [ $failed_checks -gt 0 ]; then
    echo \"❌ Pre-commit failed with $failed_checks failed checks\"
    exit 1
fi

echo \"✅ All pre-commit checks passed!\"
exit 0

#! /usr/bin/env python3
\"\"\"
Pre-commit configuration validator
Validates .pre-commit-config.yaml files against team policies to prevent outages
Version: 0.1.0
\"\"\"

import yaml
import sys
import os
from typing import List, Dict, Any

# Policy configuration
MAX_HOOK_TIMEOUT = 60  # Seconds, hooks taking longer than this are rejected
REQUIRED_CHECKS = [\"secret-scan\", \"lint\", \"type-check\"]
ALLOWED_SECRET_SCAN_TOOLS = [\"trufflehog\", \"gitleaks\"]
MAX_SECRET_SCAN_PATHS = 5

class ConfigValidationError(Exception):
    \"\"\"Custom exception for configuration validation failures\"\"\"
    pass

def load_config(config_path: str) -> Dict[str, Any]:
    \"\"\"Load and parse the pre-commit YAML config file\"\"\"
    if not os.path.exists(config_path):
        raise ConfigValidationError(f\"Config file {config_path} does not exist\")

    try:
        with open(config_path, 'r') as f:
            config = yaml.safe_load(f)
            if not isinstance(config, dict):
                raise ConfigValidationError(\"Config root must be a dictionary\")
            return config
    except yaml.YAMLError as e:
        raise ConfigValidationError(f\"Failed to parse YAML: {str(e)}\")

def validate_hooks(config: Dict[str, Any]) -> List[str]:
    \"\"\"Validate individual hook configurations\"\"\"
    errors = []
    hooks = config.get(\"hooks\", [])

    if not hooks:
        errors.append(\"No hooks defined in configuration\")
        return errors

    hook_names = [h.get(\"id\") for h in hooks if h.get(\"id\")]
    # Check required checks exist
    for required in REQUIRED_CHECKS:
        if required not in hook_names:
            errors.append(f\"Missing required check: {required}\")

    for hook in hooks:
        hook_id = hook.get(\"id\", \"unknown\")
        # Validate timeout
        timeout = hook.get(\"timeout\", 0)
        if timeout > MAX_HOOK_TIMEOUT:
            errors.append(f\"Hook {hook_id} timeout {timeout}s exceeds max {MAX_HOOK_TIMEOUT}s\")

        # Validate secret scan tools
        if \"secret\" in hook_id.lower():
            entry = hook.get(\"entry\", \"\")
            if not any(tool in entry for tool in ALLOWED_SECRET_SCAN_TOOLS):
                errors.append(f\"Hook {hook_id} uses unapproved tool: {entry}\")

            # Validate scan paths
            args = hook.get(\"args\", [])
            include_paths = [a for a in args if a.startswith(\"--include-paths\")]
            if len(include_paths) > MAX_SECRET_SCAN_PATHS:
                errors.append(f\"Hook {hook_id} has too many scan paths (max {MAX_SECRET_SCAN_PATHS})\")

    return errors

def main() -> int:
    \"\"\"Main entry point for validator\"\"\"
    if len(sys.argv) != 2:
        print(f\"Usage: {sys.argv[0]} \", file=sys.stderr)
        return 1

    config_path = sys.argv[1]
    try:
        config = load_config(config_path)
        errors = validate_hooks(config)

        if errors:
            print(\"❌ Pre-commit config validation failed:\")
            for error in errors:
                print(f\"  - {error}\", file=sys.stderr)
            return 1

        print(\"✅ Pre-commit config validation passed!\")
        return 0
    except ConfigValidationError as e:
        print(f\"❌ Validation error: {str(e)}\", file=sys.stderr)
        return 1
    except Exception as e:
        print(f\"❌ Unexpected error: {str(e)}\", file=sys.stderr)
        return 1

if __name__ == \"__main__\":
    sys.exit(main())

Pre-Commit Hook Performance: Before vs After Fix

Metric

Broken Config (1.2.1)

Fixed Config (1.2.2)

Improvement

Average hook runtime (seconds)

127

14

89% faster

False positive rate

42%

3%

39pp reduction

Commits blocked incorrectly

100% (for 2 hours)

0% (post-fix)

100% reduction

CPU usage during hook run

780% (8 cores maxed)

120% (1.2 cores)

85% reduction

Memory usage during hook run

4.2GB

320MB

92% reduction

Developer time lost per failed commit

12 minutes (debug + rerun)

45 seconds (fix and rerun)

94% faster recovery

Case Study: 50-Person Fintech Startup Monorepo

• Team size: 47 engineers (12 backend, 22 frontend, 8 mobile, 5 DevOps)
• Stack & Versions: Node.js 20.11.1, TypeScript 5.3.3, React 18.2.0, pre-commit 3.6.2, husky 9.0.11, trufflehog 3.56.0, ESLint 8.57.0, monorepo with 142 packages managed by Turborepo 1.13.0
• Problem: Pre-commit hook misconfiguration (added --regex flag to trufflehog) caused all commits to fail with exit code 1, as trufflehog scanned all generated files in node_modules and dist folders, hitting rate limits and returning false positives. For 127 minutes, 0 commits were merged, p99 CI pipeline queue time spiked to 47 minutes, and $12,400 in engineering time was lost.
• Solution & Implementation: Rolled back the pre-commit config change via git revert, then fixed the hook by removing the --regex flag, adding --entropy=False to trufflehog, adding a 30s timeout, limiting scan paths to src/ and packages/, adding ESLint caching, and implementing a pre-merge validation script (the Python validator above) to run in CI on all .pre-commit-config.yaml changes.
• Outcome: Hook runtime dropped from 127s to 14s average, false positive rate fell from 42% to 3%, incorrect commit blocks reduced to 0, saving an estimated $47k/year in lost engineering time, and p99 CI queue time returned to baseline 12 seconds within 1 hour of the fix.

3 Actionable Tips for Pre-Commit Hook Reliability

1. Always Run Hooks Locally in Dry-Run Mode Before Merging Config Changes

One of the biggest mistakes teams make is merging pre-commit configuration changes without testing them on a representative sample of commits. In our outage, the PR that introduced the broken --regex flag only ran trufflehog on a small test repo, not the full monorepo with 142 packages, generated dist folders, and 6 years of git history. Dry-run mode lets you simulate the hook run without actually modifying the working tree, so you can catch performance issues and false positives early. For pre-commit (the Python tool), use pre-commit run --dry-run; for husky-managed hooks, add a --dry-run flag to your hook scripts or use husky run --dry-run (available in husky 9.0.0+). We now require all pre-commit config PRs to include a dry-run output for a commit that touches generated files, a commit that touches sensitive files (e.g., .env.example), and a commit with a known lint error. This adds 5 minutes to PR review but has prevented 3 near-outages in the 6 months since the incident. A 2024 DevOps Research and Assessment (DORA) report found that teams who test pre-commit configs in dry-run mode have 72% fewer hook-related outages than teams that don't. The key here is to test against edge cases, not just happy paths: include large files, generated code, and binary assets in your dry-run tests to ensure the hook doesn't hang or crash. We also added a required checklist item to all pre-commit PRs: \"Dry-run tested on monorepo root, 3 different packages, and a commit with 100+ changed files.\" This simple step would have caught our --regex flag issue immediately, as the dry run would have taken over 2 minutes and scanned thousands of irrelevant files.

# Dry-run pre-commit hook on current staged changes
pre-commit run --dry-run --all-files

# Dry-run husky hook (husky 9.0.0+)
npx husky run pre-commit --dry-run

2. Implement Policy-as-Code Validation for All Hook Configurations

Manual review of pre-commit configs is error-prone, especially for large teams with frequent config changes. Our outage happened because a manual review missed that the --regex flag would cause trufflehog to scan all files in the repo, not just staged changes. Policy-as-code solves this by defining hard rules for what a valid pre-commit config looks like, then automatically enforcing those rules in CI before a config change is merged. We use the Python validator script (Code Example 3) in our GitHub Actions CI pipeline, which runs on every PR that modifies .pre-commit-config.yaml, .husky/*, or any hook script. The policy enforces rules like: maximum hook timeout of 60 seconds, required checks (secret scanning, linting, type checking), approved secret scanning tools (trufflehog or gitleaks), and a maximum of 5 scan paths per secret scanning hook. If a PR violates any of these rules, the CI check fails and blocks merge. This shifts validation left, so engineers get feedback in minutes instead of after a config is merged and breaks production. We also added a second layer of policy enforcement using Open Policy Agent (OPA) to validate that hooks don't use unapproved tools or excessive timeouts, which catches edge cases the Python script misses. Since implementing policy-as-code, we've had zero hook-related outages, and config review time has dropped from 15 minutes per PR to 2 minutes, since reviewers only need to check non-policy items like hook logic changes. For small teams, you can start with a simple shell script that checks for forbidden flags (like --regex in trufflehog) instead of a full Python/OPA setup, but the key is to automate as much as possible to avoid human error.

# GitHub Actions step to run pre-commit config validator
- name: Validate pre-commit config
  run: |
    pip install pyyaml
    python scripts/validate_precommit_config.py .pre-commit-config.yaml

3. Add Circuit Breakers and Fallbacks to Critical Hooks

Even with testing and policy-as-code, hooks can still fail due to network issues, rate limits, or tool bugs. In our outage, trufflehog hit API rate limits when scanning large commits, which caused it to return exit code 1 even when no secrets were present. Circuit breakers and fallbacks ensure that a single hook failure doesn't block all commits. For example, if secret scanning fails 3 times in a row, the hook can skip secret scanning and post a Slack alert to the DevOps team instead of blocking the commit. We implemented a simple circuit breaker in our hook script that tracks consecutive secret scan failures in a local file (~/.cache/precommit/scan_failures), and if failures exceed 3, it skips the scan and adds a warning to the commit message. We also added a fallback to gitleaks if trufflehog fails, so secret scanning still runs even if one tool is down. Another critical fallback is to allow commits with a special \"SKIP_HOOKS\" tag in the commit message for emergency fixes, but this is rate-limited to 2 per day per engineer and requires manager approval via a Slack workflow. This saved us during a recent incident where trufflehog's API was down for 30 minutes: engineers could still commit emergency fixes by adding the SKIP_HOOKS tag, and we only had 2 such commits that day, both of which were audited after the fact. The key here is to balance reliability with security: don't skip critical checks like secret scanning for non-emergency commits, but have a way to bypass hooks when tools are down to avoid another multi-hour outage. We also added metrics for hook success/failure rates to our Datadog dashboard, so we can detect when a tool is failing before it blocks all commits.

# Circuit breaker snippet for secret scanning
FAILURE_CACHE=\"$HOME/.cache/precommit/scan_failures\"
mkdir -p \"$(dirname \"$FAILURE_CACHE\")\"
failures=$(cat \"$FAILURE_CACHE\" 2>/dev/null || echo 0)

if [ \"$failures\" -ge 3 ]; then
    echo \"⚠️  Secret scan failed $failures times, skipping\"
    echo \"[SKIP_SECRET_SCAN]\" >> \"$GIT_DIR/COMMIT_MSG\"
    exit 0
fi

if ! run_secret_scan; then
    echo $((failures + 1)) > \"$FAILURE_CACHE\"
    exit 1
else
    echo 0 > \"$FAILURE_CACHE\"
fi

Join the Discussion

Pre-commit hooks are a critical part of the developer workflow, but they're also a single point of failure when misconfigured. We've shared our war story and fixes, but we want to hear from you: how does your team handle pre-commit hook reliability? What tools do you use, and what near-misses have you had?

Discussion Questions

• Will policy-as-code for pre-commit hooks become mandatory for SOC 2 compliance by 2027, as Gartner predicts?
• What is the bigger trade-off: slightly slower pre-commit hooks with full checks, or faster hooks that skip non-critical checks?
• How does pre-commit (the Python tool) compare to lefthook for large monorepos with 100+ packages, in terms of reliability and performance?

Frequently Asked Questions

How do I test pre-commit hooks on a monorepo without running them on all 142 packages?

Use the --files flag to specify a subset of files to scan, or run the hook from a specific package directory. For Turborepo monorepos, you can use turbo run pre-commit --filter=./packages/your-package to run hooks only on changed files in a specific package. We also recommend creating a test package with known edge cases (large files, generated code, secrets) to use for hook validation.

What is the maximum acceptable pre-commit hook runtime for a large team?

Based on our benchmarks and DORA research, pre-commit hooks should run in under 15 seconds for 95% of commits. Hooks taking longer than 30 seconds cause engineers to context-switch, leading to a 22% drop in productivity per the 2024 Stack Overflow Developer Survey. If your hook takes longer than 15 seconds, add caching (like ESLint's --cache flag) or split non-critical checks (like documentation generation) to a post-commit or CI pipeline instead.

Should we use pre-commit (Python) or husky for managing hooks in a Node.js monorepo?

Both tools are reliable, but husky is more tightly integrated with the Node.js ecosystem, while pre-commit has better support for multi-language hooks (Python, Go, etc.). We use both: husky to manage the hook lifecycle (installing/updating hooks) and pre-commit to define the actual hook configurations, since pre-commit's YAML config is easier to version and validate than husky's individual shell scripts. For Node-only teams, husky is simpler; for teams with mixed languages, pre-commit is more flexible.

Conclusion & Call to Action

Pre-commit hooks are a double-edged sword: they catch bugs and secrets early, but a single misconfiguration can block your entire engineering team for hours. Our 2-hour outage cost $12,400 and eroded trust in our CI pipeline, but it taught us three critical lessons: always test hook configs in dry-run mode, automate policy enforcement for all hook changes, and add circuit breakers to prevent single points of failure. If you take one thing away from this war story, it's to audit your pre-commit hooks today: check for untested flags, missing timeouts, and undefined failure modes. Run a dry-run on your last 10 commits, and if any take longer than 15 seconds, fix them immediately. Your team's productivity depends on it.

$12,400Cost of our 2-hour pre-commit hook outage (47 engineers × $220/hour)