Hardcoded secrets are the gift that keeps on giving — to attackers. API keys, tokens, database passwords, and private keys end up in repositories more often than you'd think, and once committed, they live in git history forever unless you rewrite it. Building a scanner that runs in your CI/CD pipeline catches them before they propagate.
Why Default Solutions Fall Short
Tools like git-secrets or trufflehog are useful but come with tradeoffs: they're either too slow for pre-commit hooks, or they miss context-specific patterns your codebase uses. More importantly, most teams install them and forget about them — no visibility into what's being flagged, no metrics, no alerting.
What you actually want is a lightweight scanner you control, with:
- Custom patterns for your stack (internal tokens, environment-specific formats)
- Entropy analysis for base64/hex blobs that don't match known patterns
- A clean CI integration that fails fast and reports clearly
- JSON output you can pipe into a SIEM or Slack notification
Let's build it.
The Scanner Core in Python
The foundation is a recursive file walker with regex matching against a curated pattern list.
import re
import os
import json
import math
import sys
from pathlib import Path
from dataclasses import dataclass
from typing import Optional
PATTERNS = {
"aws_access_key": r"AKIA[0-9A-Z]{16}",
"aws_secret_key": r"(?i)aws.{0,20}secret.{0,20}['\"][0-9a-zA-Z/+]{40}['\"]",
"github_token": r"ghp_[0-9a-zA-Z]{36}",
"github_classic_pat": r"github_pat_[0-9a-zA-Z_]{82}",
"stripe_secret": r"sk_(live|test)_[0-9a-zA-Z]{24,}",
"stripe_publishable": r"pk_(live|test)_[0-9a-zA-Z]{24,}",
"sendgrid_api_key": r"SG\.[0-9a-zA-Z\-_]{22}\.[0-9a-zA-Z\-_]{43}",
"jwt_token": r"eyJ[a-zA-Z0-9_-]{10,}\.[a-zA-Z0-9_-]{10,}\.[a-zA-Z0-9_-]{10,}",
"private_key_header": r"-----BEGIN (RSA |EC |OPENSSH )?PRIVATE KEY-----",
"generic_password": r"(?i)(password|passwd|pwd)\s*[:=]\s*['\"][^'\"]{8,}['\"]",
"generic_api_key": r"(?i)(api[_-]?key|apikey)\s*[:=]\s*['\"][^'\"]{16,}['\"]",
"db_connection_str": r"(?i)(mongodb|postgres|mysql|redis)://[^@\s]+:[^@\s]+@",
}
SKIP_EXTENSIONS = {".png", ".jpg", ".jpeg", ".gif", ".svg", ".ico",
".pdf", ".zip", ".tar", ".gz", ".bin", ".exe", ".lock"}
SKIP_DIRS = {".git", "node_modules", "__pycache__", ".venv", "vendor", "dist", "build"}
@dataclass
class Finding:
file: str
line: int
pattern_name: str
matched_text: str
entropy: Optional[float] = None
def shannon_entropy(data: str) -> float:
if not data:
return 0.0
freq: dict[str, int] = {}
for c in data:
freq[c] = freq.get(c, 0) + 1
return -sum(
(count / len(data)) * math.log2(count / len(data))
for count in freq.values()
)
def scan_file(path: Path, compiled: dict) -> list[Finding]:
findings = []
try:
text = path.read_text(errors="replace")
except (PermissionError, IsADirectoryError):
return findings
for lineno, line in enumerate(text.splitlines(), 1):
for name, pattern in compiled.items():
match = pattern.search(line)
if match:
matched = match.group(0)
entropy = shannon_entropy(matched) if len(matched) > 12 else None
findings.append(Finding(
file=str(path),
line=lineno,
pattern_name=name,
matched_text=matched[:80],
entropy=round(entropy, 2) if entropy else None,
))
return findings
def scan_directory(root: str) -> list[Finding]:
compiled = {name: re.compile(pat) for name, pat in PATTERNS.items()}
all_findings: list[Finding] = []
for dirpath, dirnames, filenames in os.walk(root):
dirnames[:] = [d for d in dirnames if d not in SKIP_DIRS]
for filename in filenames:
path = Path(dirpath) / filename
if path.suffix.lower() in SKIP_EXTENSIONS:
continue
all_findings.extend(scan_file(path, compiled))
return all_findings
if __name__ == "__main__":
root = sys.argv[1] if len(sys.argv) > 1 else "."
findings = scan_directory(root)
output = [vars(f) for f in findings]
print(json.dumps(output, indent=2))
sys.exit(1 if findings else 0)
The exit code matters: sys.exit(1) on findings makes your CI pipeline fail automatically without any additional configuration.
Entropy-Based Detection
Regex patterns catch known formats. High-entropy strings catch everything else — random tokens, secrets generated by internal systems, base64-encoded keys that don't match a known provider's format.
Shannon entropy measures randomness. A typical English word has entropy around 3–4 bits per character. A properly generated API key or secret usually lands above 4.5.
Add a second pass that flags high-entropy strings in common assignment contexts:
import re
import math
HIGH_ENTROPY_PATTERN = re.compile(
r"(?ix)"
r"(?:secret|token|key|password|credential|auth)[^\n]{0,10}"
r"['\"\s=:]+"
r"([a-zA-Z0-9+/=_\-]{20,})"
)
ENTROPY_THRESHOLD = 4.5
def scan_for_high_entropy(text: str, filename: str) -> list[dict]:
results = []
for lineno, line in enumerate(text.splitlines(), 1):
for match in HIGH_ENTROPY_PATTERN.finditer(line):
candidate = match.group(1)
entropy = shannon_entropy(candidate)
if entropy >= ENTROPY_THRESHOLD:
results.append({
"file": filename,
"line": lineno,
"pattern_name": "high_entropy_string",
"matched_text": candidate[:60],
"entropy": round(entropy, 2),
})
return results
Run this after the pattern pass. A combined signal — pattern match and high entropy — gives you very high confidence it's a real secret rather than a false positive like a CSS hash or a long test fixture ID.
Wiring It Into CI/CD
GitHub Actions
Create .github/workflows/secrets-scan.yml:
name: Secrets Scan
on:
push:
branches: ["**"]
pull_request:
jobs:
scan:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
- uses: actions/setup-python@v5
with:
python-version: "3.12"
- name: Run secrets scanner
run: |
python scripts/scan_secrets.py . 2>&1 | tee scan-results.json
- name: Upload results
if: always()
uses: actions/upload-artifact@v4
with:
name: secrets-scan-results
path: scan-results.json
fetch-depth: 0 fetches full history, which you need if you want to diff only changed files across branches. For a full-repo scan on every push, the default shallow clone works fine.
For GitLab CI, the equivalent is a job in .gitlab-ci.yml that runs the same script with allow_failure: false and uploads the JSON as an artifact.
Reducing False Positives Without Losing Coverage
The biggest operational problem with secrets scanners is alert fatigue. If your scanner fires on every test fixture or example config, engineers start ignoring it — which is worse than having no scanner at all.
Allowlist by file path, not pattern. Adding a pattern to an ignore list silences it everywhere. Instead, allow specific directories: tests/fixtures/, docs/examples/, .env.example. Your scanner stays strict on production code while being quiet on intentional examples.
Require entropy threshold for generic patterns. The generic_password and generic_api_key patterns above will generate noise if used alone. Only surface them when the matched value also has entropy above 3.8.
Be strict about format. For AWS keys, verify that AKIA is followed by exactly 16 uppercase alphanumeric characters. Tightening the regex almost eliminates false positives for known service formats.
Diff-only mode for PRs, full-scan on main. On pull requests, scan only the changed lines. On pushes to main, scan the full tree. This keeps PR checks fast without reducing coverage where it matters most.
For teams managing multiple services, pairing your scanner with a standardized security hardening checklist makes it easier to enforce consistent configuration across repos — same patterns, same thresholds, same allowlist policy applied everywhere.
The Takeaway
A secrets scanner in CI is table stakes for any production system. The implementation above is roughly 100 lines of Python with zero dependencies beyond the standard library — there is no excuse not to run it.
The key design decisions: explicit pattern names (so findings are actionable, not just "found something"), entropy scoring (so you catch what regex misses), and a non-zero exit code on findings (so CI actually fails instead of just logging). Start with the pattern list here, extend it for your internal token formats, and wire it into your pipeline before the next deploy.
I run AYI NEDJIMI Consultants, a cybersecurity consulting firm. We publish free security hardening checklists — PDF and Excel.
Top comments (0)