DEV Community

Cover image for I Built an LLM That Reads Compliance Regulations and Blocks Non-Compliant Code Deployments IEEE Published the Results
Ajay Devineni
Ajay Devineni

Posted on

I Built an LLM That Reads Compliance Regulations and Blocks Non-Compliant Code Deployments IEEE Published the Results

My paper on LLM-based regulatory compliance automation for financial DevOps was published at IEEE ICCBI 2026. The core idea sounds almost too good: an LLM reads regulatory documents (Basel III, GDPR, PCI-DSS), extracts compliance rules, and blocks non-compliant code from deploying. Here's how we actually built it, and where the hard problems are.

Why Manual Compliance Is Broken

Financial institutions face regulatory requirements that are:

  • Written in legal prose, not machine executable rules
  • Updated frequently — GDPR guidance evolves, Basel capital requirements change, new national AI regulations appear
  • Jurisdictionally contradictory a system compliant with GDPR may violate a local data sovereignty law in a different market

The standard approach: a compliance team manually translates regulations into a static checklist, developers manually verify their changes against the checklist, and an audit team manually verifies everything before regulators arrive.

This doesn't work at DevOps velocity. Code ships 10–50 times per day. The compliance team can't review every commit.

The LLM Solution — And Its Problems

The framework uses an LLM to do what compliance officers do: read regulatory text and extract actionable rules. But naively prompting an LLM with "here is GDPR article 25, what does it require?" creates a dangerous new problem: hallucination in compliance contexts.

A compliance system that confidently generates incorrect requirements is worse than no compliance system.

Here's how the framework addresses this:

RAG-Backed Rule Generation 

from agentsre.compliance import RegulatoryRuleExtractor

extractor = RegulatoryRuleExtractor(
    llm_backend='bedrock',          # Amazon Bedrock for financial services
    model_id='anthropic.claude-3-5-sonnet',
    rag_source_bucket='s3://compliance-docs/regulations/',
    citation_required=True,         # EVERY rule must cite source text
    confidence_threshold=0.85       # Low-confidence rules go to human review
)

# Extract rules from regulatory document
rules = extractor.extract(
    document='gdpr_article_25_data_protection_by_design.pdf',
    scope=['data_storage', 'encryption', 'access_control', 'retention']
)

# Returns structured rules WITH citations
# {
#   'rule_id': 'GDPR-25-1',
#   'requirement': 'Personal data must be encrypted at rest using AES-256 or equivalent',
#   'source_citation': 'GDPR Article 25(1): "...appropriate technical measures..."',
#   'confidence': 0.94,
#   'infrastructure_targets': ['S3', 'RDS', 'EBS', 'DynamoDB']
# }
Enter fullscreen mode Exit fullscreen mode

Key design decision: the LLM never operates from memory. It always re-reads the source regulatory document through RAG before generating a rule. This eliminates the most dangerous hallucination vector — stale or fabricated regulatory requirements.

CI/CD Gate Integration 

# .github/workflows/compliance-gate.yml (conceptual)
name: Compliance Gate

on: [pull_request]

jobs:
  compliance-check:
    runs-on: ubuntu-latest
    steps:
      - name: Evaluate Compliance Impact
        run: |
          agentsre-compliance evaluate \
            --diff ${{ github.event.pull_request.diff_url }} \
            --rule-registry $COMPLIANCE_REGISTRY_URL \
            --jurisdictions "US,EU,UK" \
            --failure-mode strict \
            --output compliance-report.json

      - name: Block on Non-Compliance
        if: steps.compliance-check.outputs.compliant == 'false'
        run: |
          echo "::error::Compliance violations detected. See compliance-report.json"
          cat compliance-report.json
          exit 1
Enter fullscreen mode Exit fullscreen mode

The gate evaluates the code diff — not just the final state — because compliance violations are often introduced in specific changes and obscured in the final artifact.

Multi-Jurisdictional Conflict Resolution 

from agentsre.compliance import JurisdictionResolver

resolver = JurisdictionResolver(
    jurisdictions=['GDPR', 'CCPA', 'PDPA_SG', 'LGPD_BR'],
    conflict_strategy='strictest_wins',
    human_escalation_threshold='direct_conflict'
)

# When GDPR requires deletion within 30 days
# and local law requires retention for 7 years:
# → Escalate to human policy decision, don't resolve autonomously
resolution = resolver.evaluate(rule_set)
Enter fullscreen mode Exit fullscreen mode

strictest_wins for ambiguity, human_escalation for direct conflict. This is the right failure mode for regulated systems.

Hard Problems We Haven't Fully Solved

Regulatory update propagation: When a regulatory body issues guidance that amends an existing regulation without changing the source document, the RAG system won't catch it. We handle this with a human-curated "regulatory update feed" that triggers rule re-extraction when new guidance is published.

Legacy system integration: Some financial systems expose compliance-relevant configurations only through mainframe JCL or COBOL report outputs. Parsing these into a format the framework can evaluate required custom adapters — not generalizable.

False positive rate in blocking mode: Early versions of the framework blocked 30% of legitimate deployments with false positive compliance violations. After tuning confidence thresholds and adding a "remediation suggestion" layer (not just FAIL, but "here's how to fix it"), false positive rates dropped to acceptable levels.

Results

Benchmarked against manual compliance processes and rule-based automated tools:

  • Compliance coverage: LLM framework covers regulatory requirements that static rule engines miss — particularly ambiguous or cross-referenced clauses
  • Audit preparation time: Reduced from weeks to hours through continuous audit trail generation
  • Regulatory update lag: Near-zero — framework re-extracts rules from updated source documents automatically
  • CI/CD throughput impact: Minimal — async validation for low-risk changes, synchronous blocking only for high-risk patterns

Try It 

pip install agentsre
Enter fullscreen mode Exit fullscreen mode

The agentsre.compliance subpackage (in development) implements the rule extraction and CI/CD gate components. Contributions welcome.

GitHub: github.com/Ajay150313/agentsre

Paper: IEEE ICCBI 2026, Paper ID ICCBI-870 → https://ieeexplore.ieee.org/abstract/document/11619889

Google Scholar: scholar.google.com/citations?user=AyVSzecAAAAJ

If you're implementing compliance automation in a regulated industry — financial services, healthcare, government I'd genuinely like to hear what approaches you've tried. Particularly interested in how teams handle LLM confidence calibration for high-stakes decisions.

Top comments (0)