We've been building Spidercob, an enterprise DLP platform. The hardest problem was never detecting sensitive data it was not detecting things that aren't sensitive.
Regex-based scanners are easy to write. A pattern for AWS access keys takes five minutes. The problem is that same pattern fires on any 20-character uppercase string in your codebase test fixtures, documentation examples, README placeholders, all flagged as critical.
After six months of tuning in production we extracted our detection engine into a standalone library:
dlp-patterns. Here's what actually works.
## The Three Layers
### 1. Validators
The first thing we added was format validation after the regex match. A regex is a necessary condition, not a sufficient one.
Credit cards - every match runs through Luhn's algorithm. A transposed digit fails Luhn. This alone cuts credit card false positives by roughly 90%.
python
def _luhn_check(number: str) -> bool:
digits = [int(d) for d in number if d.isdigit()]
odd_digits = digits[-1::-2]
even_digits = digits[-2::-2]
total = sum(odd_digits)
for d in even_digits:
total += sum(divmod(d * 2, 10))
return total % 10 == 0
SSNs — the IRS publishes rules about which area codes are invalid. 000, 666, and 900-999 are all unassigned. We reject those before surfacing a finding.
JWTs — we base64-decode the header and payload and verify they parse as valid JSON with expected fields (alg, typ). A random string that happens to match the JWT pattern fails this check immediately.
2. Entropy Gating
This is the biggest lever we found. Low-entropy strings are almost never real secrets.
Shannon entropy measures how random a string is. A real AWS secret key looks like noise. A test placeholder like AKIAIOSFODNN7EXAMPLE —
ironically AWS's own documentation example — has lower entropy because it contains recognisable English words.
We use a sliding window approach:
import math
def _shannon_entropy(s: str) -> float:
if not s:
return 0.0
freq = {}
for c in s:
freq[c] = freq.get(c, 0) + 1
length = len(s)
return -sum(
(count / length) * math.log2(count / length)
for count in freq.values()
)
def _passes_entropy_gate(value: str, min_entropy: float = 3.5) -> bool:
if _shannon_entropy(value) < min_entropy:
return False
# Sliding window catches strings like "aaabbbccc" that fool overall entropy
window = 8
for i in range(len(value) - window + 1):
chunk = value[i:i + window]
if len(set(chunk)) < 4:
return False
return True
This single check eliminates the vast majority of generic secret false positives. Repeated characters, dictionary words, sequential patterns —
all rejected without touching a single regex.
3. Context Scoring
The last layer doesn't reject findings outright — it scores them.
Every finding gets a context_score between 0.0 and 1.0 based on the words surrounding it. We scan ±100 characters around each match:
Boost keywords — score goes up: production, prod, secret, credential, deploy, live, api_key
Penalty keywords — score goes down: example, placeholder, test, sample, dummy, fake, replace
BOOST_KEYWORDS = {"production", "prod", "secret", "credential", "deploy", "live"}
PENALTY_KEYWORDS = {"example", "placeholder", "test", "sample", "dummy", "fake"}
def _context_score(text: str, match_start: int, match_end: int) -> float:
window_start = max(0, match_start - 100)
window_end = min(len(text), match_end + 100)
context = text[window_start:window_end].lower()
words = set(context.split())
score = 0.5 # neutral baseline
score += len(words & BOOST_KEYWORDS) * 0.15
score -= len(words & PENALTY_KEYWORDS) * 0.20
return max(0.0, min(1.0, score))
You can threshold on context_score in your pipeline. In CI, only block on findings with context_score > 0.7. In a compliance scan, report
everything but prioritise high-score findings.
Required Context Keywords
Some patterns are too ambiguous to fire without corroborating evidence.
ICD-10 codes like E11.9 are a good example. The pattern [A-Z]\d{2}\.\d{1,4} matches thousands of strings that aren't medical codes. We only fire
the ICD-10 pattern when medical keywords appear nearby:
diagnosis, icd, icd-10, condition, disease, disorder,
patient, clinical, medical, treatment
Without one of those words in context the pattern stays silent. This approach works well for any structurally ambiguous pattern — Telegram bot
tokens, NPI numbers, DEA registration numbers.
Using It
import dlp_patterns
result = dlp_patterns.scan(text)
result.has_findings # bool
result.highest_severity # "CRITICAL" | "HIGH" | "MEDIUM" | "LOW" | None
result.critical # list[Finding]
f = result.critical[0]
f.type # "aws_access_key"
f.value # masked value
f.context_score # 0.0 – 1.0
f.context_keywords_found # ["production", "deploy"]
# Redact in one line
clean = dlp_patterns.redact(text)
# Secrets only — skip PII, faster for CI
result = dlp_patterns.scan(code, secrets_only=True)
CLI for pipelines:
# Fail CI on CRITICAL findings
dlp-scan -secrets-only src/
# Redact logs before shipping to external services
cat app.log | dlp-scan -redact > safe.log
# JSON output for SIEM integration
dlp-scan -json document.txt | tee dlp-report.json
GitHub Actions — use the official action:
- name: DLP Secret Scan
uses: spidercob/dlp-scan-action@v1
with:
secrets-only: 'true'
fail-on: 'critical'
50+ Pattern Categories
┌────────────────┬────────────────────────────────────────────────────────────────────────┐
│ Category │ Patterns │
├────────────────┼────────────────────────────────────────────────────────────────────────┤
│ Secrets │ AWS, GitHub PATs, Stripe, SendGrid, Slack, Twilio, Discord, Google API │
├────────────────┼────────────────────────────────────────────────────────────────────────┤
│ Infrastructure │ DB connection strings, hardcoded passwords, Docker registry auth │
├────────────────┼────────────────────────────────────────────────────────────────────────┤
│ Crypto keys │ RSA / EC / SSH / PGP private keys, X.509 certs │
├────────────────┼────────────────────────────────────────────────────────────────────────┤
│ PII │ SSN, credit cards, email, US phone, passport, date of birth │
├────────────────┼────────────────────────────────────────────────────────────────────────┤
│ Healthcare │ Medical record numbers, ICD-10 codes, NPI, DEA numbers │
├────────────────┼────────────────────────────────────────────────────────────────────────┤
│ Crypto wallets │ Bitcoin addresses, Ethereum addresses │
└────────────────┴────────────────────────────────────────────────────────────────────────┘
Install
pip install dlp-patterns
Zero dependencies. Python 3.9+. Apache 2.0.
Source: https://github.com/SpiderCob/dlp-patterns
GitHub Action: https://github.com/SpiderCob/dlp-scan-action
If you're scanning at scale and need dashboards, audit logs, ICAP proxy integration, or Gmail/Slack connectors — that's what
https://spidercob.com is built for.
Top comments (0)