DEV Community: Sanjoy Ghosh

How GAUNTLEX Gates HIPAA/FINRA Compliance in CI

Sanjoy Ghosh — Thu, 16 Jul 2026 16:27:40 +0000

The compliance gap nobody's pricing in yet

A team building a healthcare API asks an AI coding assistant to implement a patient records endpoint. The assistant produces working code — clean, well-structured, passes review. Six months later, a HIPAA audit finds the endpoint returns more PHI fields than the request actually needed. Nobody wrote a test for "is this response over-broad." Nobody thought to.

That's not a hypothetical. It's the specific, named failure mode in GAUNTLEX's own HIPAA policy playbook: "PHI Disclosure — Over-Broad API Response." I want to walk through how GAUNTLEX turns regulatory requirements like this into something that actually runs in CI, instead of something a compliance team discovers during an audit that's already too late.

Domains, not generic scans

GAUNTLEX ships five compliance domain playbooks out of the box: OWASP Top 10, HIPAA, FINRA, PCI DSS, and SOC 2 (NIST SSDF and OWASP API Security are available as installable extensions). Each one is a curated set of attack scenarios specific to that domain's actual failure modes — not a generic vulnerability scanner with a different label stapled on.

The HIPAA playbook's nine scenarios include things like "Emergency Access — Hardcoded Override Credentials" and "PHI Integrity — Missing Tamper Detection" — the kind of finding a generic SAST tool has no vocabulary for, because it's not a code pattern, it's a regulatory failure mode. The FINRA playbook is a different nine entirely: "SEC 17a-4 — Non-WORM Record Storage Race Condition," "AML — Structuring Detection Bypass," "Best Execution — Pricing Calculation Error" — scenarios written for people who've actually read those regulations, not reverse-engineered from a CVE database.

gauntlex run --issue patient_api_spec.md --mode standard --domain hipaa

Every finding traces to a control

This is the part that actually matters for a compliance review. Every finding GAUNTLEX produces carries a CWE tag and maps to real control frameworks:

CONTROL_MAPPINGS = {
    "NIST_SSDF": ["RV.2.2", "RV.3.1", "PW.8.1"],
    "OWASP_SAMM": ["Verification/Security-Testing/2"],
    "SOC2_CC": ["CC7.1", "CC8.1"],
    "ISO_27001": ["A.14.2.8", "A.14.2.9"],
}

That's not decoration. It's the difference between handing an auditor "we ran a security tool" and handing them a report where every finding traces to the specific control it violates — the artifact a compliance reviewer actually needs, in the format they actually work in.

The gate, not a suggestion

Here's where it stops being a reporting tool and becomes an enforcement mechanism. GAUNTLEX runs in CI with a configurable minimum Adversarial Resilience Score — 0.80 by default — and fail_open: false. Below that threshold, the merge is blocked. Not a Slack notification to review later. The same mechanism that blocks a merge on a failing test suite, applied to compliance-relevant security posture.

gate:
  minimum_ars: 0.80
  fail_open: false

That single config line is the actual point of this whole system: compliance testing that happens before code ships, gating the same pipeline everything else already gates, instead of a quarterly audit finding what a machine could have caught on day one.

Try it

pip install gauntlex-ai
gauntlex policy list                    # see all available domains
gauntlex run --issue your_spec.md --domain hipaa --mode quick

gauntlex audit lists every past run with its full compliance control mapping over a configurable window — useful the next time an auditor asks "show me your security testing evidence" and you'd rather hand over a report than a explanation.

Repo, MCP integration, and the full domain list: github.com/sanjoy1234/gauntlex. If you're building in a regulated space and want to see what GAUNTLEX flags in your own spec, I'd genuinely like to know — open a discussion.

The Adversarial Resilience Score: A New Metric for AI-Generated Code

Sanjoy Ghosh — Tue, 14 Jul 2026 02:49:44 +0000

The problem with "this AI writes secure code"

Every AI coding tool now claims some flavor of security-awareness. Almost none of them will tell you, in a number, how resilient the code it just wrote actually is — and fewer still will let you verify that number yourself after the fact.

That's the gap I built the Adversarial Resilience Score (ARS) to close. It's the core metric behind GAUNTLEX, and I want to walk through exactly how it's computed — not the marketing version, the actual formula — because a security metric nobody can audit isn't a metric, it's a claim.

The formula

ARS = Σ(attack_scores) / N

Where N is the number of adversarial attacks fired at the generated code for a given run (5 in quick mode, 20 in standard, 50 in thorough), and each individual attack_score is one of exactly three values:

1.0 — mitigated. The attack was fully defended against.
0.5 — partial. The defense exists but is bypassable or incomplete.
0.0 — missed. No defense at all; the attack succeeded cleanly.

Each attack is scored independently by GAUNTLEX's Arbiter — a separate model call that renders a verdict (mitigated / partial / missed) plus a one-line reason, so every score comes with an explanation attached, not just a number. ARS is the mean of those scores, so it lands somewhere in [0.0, 1.0].

Why an average, not a pass rate

This is the detail I think matters most and gets glossed over the most. ARS is not "percentage of attacks blocked." A pass/fail count throws away information — it treats a defense that's 90% there the same as no defense at all, and it treats a narrowly-avoided bypass the same as an attack that never had a chance. Averaging continuous per-attack scores keeps that gradient. A codebase that mitigates most attacks cleanly but has two partially-bypassable defenses scores differently — and more informatively — than one that fully blocks some attacks and fully fails others, even if the raw counts look similar.

It also means ARS resists gaming by test selection. You can't inflate it by throwing in a pile of trivially-blocked attacks, because a single fully-missed attack against something serious (say, an auth bypass) pulls the average down hard regardless of how many easy ones surround it.

The gate, not a suggestion

GAUNTLEX runs in CI with a configurable minimum ARS — 0.80 by default. If a run scores below that threshold, the build fails. Not a warning, not a Slack notification to review later — the merge is blocked, the same way a failing test suite blocks a merge. fail_open defaults to false: if GAUNTLEX itself can't complete a run for some reason, that's a failure state too, not a silent pass.

The reasoning: a security score that's advisory gets ignored under deadline pressure, every time. A score that gates the merge doesn't have that problem.

Domain-specific, not generic

The 5/20/50 attacks per run aren't generic fuzzing — they're drawn from policy playbooks scoped to a specific regulatory or security domain: OWASP Top 10, HIPAA, FINRA, PCI DSS, SOC 2 (with NIST SSDF and OWASP API Security available as installable extensions). Each domain playbook is a curated set of scenarios specific to that domain's actual failure modes — HIPAA's playbook probes PHI-handling failure modes, FINRA's probes the record-retention and audit-trail patterns those regulations actually care about, not a repackaged OWASP list with a different label on it.

Every finding GAUNTLEX produces also carries a CWE tag and maps to control frameworks (NIST SSDF, OWASP SAMM, SOC 2, ISO 27001) — so a report isn't just "here's a score," it's something a compliance reviewer can actually trace back to a specific control.

Tamper-evident by construction

A self-reported security score is worth exactly as much as the trust you place in whoever generated it — which is close to zero for a tool that hasn't earned it yet. So every GAUNTLEX report includes a SHA-256 hash computed over the ordered array of attack results. Anyone can run gauntlex verify <run_id> and confirm the report they're looking at hasn't been edited after the fact. It doesn't prove the score is good — it proves the score you're reading is the score that was actually computed, not a touched-up version of it.

That distinction — provable versus claimed — is the whole design philosophy behind GAUNTLEX, and it's why I built ARS as an auditable formula instead of a black-box "security rating."

Try it

pip install gauntlex-ai
gauntlex run --spec your_spec.md --mode quick

Quick mode runs 5 attacks in under a minute and prints an ARS for whatever spec you point it at. Full writeup, the compliance domain list, and the MCP server integration (works inside Claude Code, Cursor, Windsurf, Zed) are in the repo and the Deep Dive doc.

If you try it against something you've built, I'd genuinely like to know what it finds — that's what Discussions is for.

Why I Built an Adversarial Co-Generation Engine

Sanjoy Ghosh — Mon, 13 Jul 2026 03:35:53 +0000

I spent a chunk of last year around legacy modernization work — the kind of project where a bank or an insurer is taking twenty years of accumulated code and rebuilding it as modern services, one system at a time. Every one of those systems starts the same way: a PRD or a requirements document says what the business needs, that gets translated into a spec precise enough for an AI to implement, and eventually someone tests what came out.

What struck me, watching this happen at scale, wasn't that the code was bad. It was that nobody was testing the thing that actually determined whether the code would be bad: the spec itself — the technical description handed to the model, not the PRD that motivated it. Every security tool I looked at — SAST scanners, DAST tools, even the AI coding assistants themselves — waited until an implementation existed before doing anything adversarial. Attack the code, once it's there. That's the whole industry's model, and it's worked fine for forty years because the volume was always survivable. A team ships a handful of PRs a week, a human reviews them, and eventually a pentest catches whatever slipped through.

That math falls apart at modernization scale. When you're regenerating a few million lines of code, you're also generating a few thousand specs, faster than any review process was ever built to absorb. Testing after the fact doesn't just get slower under that load — it quietly stops happening, spec by spec, until the aggregate exposure is enormous and nobody can point to when it happened.

So I built GAUNTLEX to test the thing that happens before the code does: the spec.

This is also where I want to be precise about a word that gets overloaded. "Spec-driven development" — the broader industry shift toward writing structured, agent-facing specs instead of prompting an AI free-form — is exactly the world GAUNTLEX lives in. But a spec (what to build, precise enough for a model to implement) and a PRD or requirements doc (why it's needed) aren't the same artifact, even though people often use "spec" loosely to mean both. GAUNTLEX treats them as two separate inputs on purpose: build intent from the spec itself, business intent pulled in separately from wherever your team actually tracks the why — Jira, Confluence, Aha! — via what the product calls the Intent Adapter. The attack surface gets reasoned from both, not just one.

How it actually works

Two agents run on the same specification, at the same time. A Builder implements it. A Breaker attacks it, reading only the spec, with no visibility into what the Builder wrote. Neither one waits on the other. By the time the Builder has working code, the Breaker already has a full adversarial test suite derived from the same intent.

A third agent, the Arbiter, scores every attack against whatever the Builder actually produced — mitigated, partially mitigated, or missed entirely — and that produces a single number, the Adversarial Resilience Score. Fall below the threshold your team sets (0.80 by default) and the merge is blocked automatically, the same way a failing test suite blocks one today. It isn't a new discipline. It's the same discipline, pointed at a gap nobody was covering.

gauntlex run --issue SPEC.md --intent PROJ-123 --mode standard

What actually comes out of it

Every finding is tied to a real CWE, but more importantly, to the compliance controls an auditor is actually going to ask about — NIST SSDF, OWASP SAMM, SOC 2, ISO 27001. A CWE number is useful to an engineer. It isn't what satisfies a compliance review, and I wanted the report to be useful to both people in that conversation, not just one. Five regulated-industry playbooks — HIPAA, FINRA, PCI DSS, SOC 2, OWASP Top 10 — ship today, each mapped to the actual regulation it's checking against, not a generic scan wearing an industry label.

If you want to try it

gauntlex setup
gauntlex run --issue examples/demo_issue.md --mode quick --pretty

It's free, MIT licensed, and works with whichever model provider you already use, including fully local through Ollama if you can't send code to an external API at all.

I'd genuinely like to hear what it catches in your own repos That's what the GitHub Discussions are for.

github.com/sanjoy1234/gauntlex