Every enterprise security team is dealing with the same problem: employees are using ChatGPT with company data, and IT has no visibility, no control, and no audit trail.
Banning doesn't work. Employees use the tools that make them productive. If you ban ChatGPT without providing a better alternative, you create a two-tier workforce — the rule-followers who fall behind, and the rule-breakers who deliver.
The actual solution: build an internal AI assistant that is more useful to employees than the public tools they're currently using, deployed with the security controls that make your CISO comfortable.
Here's the full technical architecture.
Why Internal AI Beats Public AI for Employees
ChatGPT answers general questions from general training data. An internal AI answers company-specific questions from company-specific data.
When an employee asks ChatGPT, "What's our refund policy?", they get a generic answer. When they ask an internal AI trained on your actual refund policy docs, they get the exact policy with the relevant exceptions.
This is why employees switch voluntarily. Not policy compliance — better tooling. And better tooling that doesn't require them to paste confidential data into an external service.
The Core Architecture
Employee Query
│
▼
┌─────────────────────────────────┐
│ Company SSO / Auth Layer │ ← SAML, OAuth2, Azure AD
│ (only authenticated users) │
└─────────────────────────────────┘
│
▼
┌─────────────────────────────────┐
│ PII Detection & Redaction │ ← Before query leaves internal network
│ (scan for names, IDs, PHI) │
└─────────────────────────────────┘
│ (clean query)
▼
┌─────────────────────────────────┐
│ Internal RAG Pipeline │
│ ├── Vector search (internal docs)│
│ ├── Keyword search (policies) │
│ └── Structured data lookup │
└─────────────────────────────────┘
│ (relevant context)
▼
┌─────────────────────────────────┐
│ LLM Inference │ ← Hosted internally OR private API
│ (with retrieved context) │ ← Never raw company data to external
└─────────────────────────────────┘
│
▼
┌─────────────────────────────────┐
│ Audit Logging │ ← User ID, query hash, response,
│ │ timestamp, data sources accessed
└─────────────────────────────────┘
│
▼
Response to Employee
1. Auth Integration (Non-Negotiable)
The AI assistant must authenticate through your existing identity provider — no separate accounts, no shared credentials.
from fastapi import FastAPI, Depends, HTTPException
from fastapi. security import OAuth2AuthorizationCodeBearer
import jwt
app = FastAPI()
oauth2_scheme = OAuth2AuthorizationCodeBearer(
authorizationUrl="https://login.microsoftonline.com/tenant/oauth2/v2.0/authorize",
tokenUrl="https://login.microsoftonline.com/tenant/oauth2/v2.0/token"
)
async def get_current_user(token: str = Depends(oauth2_scheme)):
try:
payload = jwt.decode(token, options={"verify_signature": False})
user_id = payload.get("oid") # Azure AD object ID
department = payload.get("department", "unknown")
return {"user_id": user_id, "department": department}
except Exception:
raise HTTPException(status_code=401, detail="Invalid authentication")
@app.post("/query")
async def process_query(
query: str,
current_user: dict = Depends(get_current_user)
):
# Only authenticated users reach this point
# User context is available for permission-scoping
return await handle_query(query, current_user)
2. PII Detection Before Any External Call
If you're using a hosted LLM API (OpenAI, Anthropic, etc.), scan every query for PII before it leaves your network. For regulated industries, this is mandatory.
import re
from dataclasses import dataclass
from typing import List
@dataclass
class PIIDetectionResult:
has_pii: bool
pii_types: List[str]
redacted_text: str
class PIIDetector:
PATTERNS = {
"email": r'\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b',
"phone": r'\b(\+\d{1,2}\s?)?\(?\d{3}\)?[\s.-]?\d{3}[\s.-]?\d{4}\b',
"ssn": r'\b\d{3}-\d{2}-\d{4}\b',
"credit_card": r'\b\d{4}[\s-]?\d{4}[\s-]?\d{4}[\s-]?\d{4}\b',
"patient_id": r'\b(MRN|DOB|Patient\s*ID)[:\s]*\S+', # Healthcare-specific
}
def detect_and_redact(self, text: str) -> PIIDetectionResult:
redacted = text
found_types = []
for pii_type, pattern in self.PATTERNS.items():
matches = re.findall(pattern, text, re.IGNORECASE)
if matches:
found_types.append(pii_type)
redacted = re.sub(pattern, f"[{pii_type.upper()}_REDACTED]",
redacted, flags=re.IGNORECASE)
return PIIDetectionResult(
has_pii=bool(found_types),
pii_types=found_types,
redacted_text=redacted
)
# Usage: scan every query before external API call
detector = PIIDetector()
result = detector.detect_and_redact(user_query)
if result.has_pii:
# Either block the query, use a redacted version, or route to on-premise model
query_to_llm = result.redacted_text
log_pii_detection(current_user, result.pii_types)
else:
query_to_llm = user_query
3. Department-Scoped RAG
Employees should only get answers from documents they're authorised to access. A sales employee shouldn't retrieve HR disciplinary records. A junior employee shouldn't retrieve board-level financial projections.
class ScopedVectorStore:
def __init__(self, base_store):
self.store = base_store
def search(self, query: str, user: dict, k: int = 5):
# Build access filter based on user's department and role
access_filter = self._build_access_filter(user)
# Only retrieve documents the user is authorised to see
return self.store.similarity_search(
query,
k=k,
filter=access_filter
)
def _build_access_filter(self, user: dict) -> dict:
# Documents tagged with department restrictions
# are only returned to users in that department
return {
"$or": [
{"access": "all"}, # Public company docs
{"access": user["department"]}, # Dept-specific docs
{"access": f"role:{user.get('role', 'employee')}"} # Role-based
]
}
4. Audit Logging (Required for Compliance)
Every query must be logged — user, timestamp, query (or hash), data sources accessed, response. This creates the audit trail that HIPAA, GDPR, and SOC 2 require.
import hashlib
import json
from datetime import datetime
class AuditLogger:
def __init__(self, log_store):
self.store = log_store
def log_query(self, user: dict, query: str,
sources: list, response: str, pii_detected: bool):
entry = {
"timestamp": datetime.utcnow().isoformat(),
"user_id": user["user_id"],
"department": user["department"],
"query_hash": hashlib.sha256(query.encode()).hexdigest(),
# Store hash, not raw query — protects employee privacy
# while maintaining audit trail
"sources_accessed": [s["doc_id"] for s in sources],
"pii_detected": pii_detected,
"response_length": len(response),
"session_id": user.get("session_id")
}
self.store.append(entry)
def get_compliance_report(self, start_date: str, end_date: str) -> dict:
"""Produces the report your compliance team will request"""
entries = self.store.get_range(start_date, end_date)
return {
"total_queries": len(entries),
"unique_users": len(set(e["user_id"] for e in entries)),
"pii_detection_events": sum(1 for e in entries if e["pii_detected"]),
"top_document_categories": self._aggregate_sources(entries)
}
The 3-Week Build Sequence
Week 1: Auth integration + document ingestion + basic RAG pipeline. First queries will be working by the end of the week.
Week 2: PII detection + department-scoped access + audit logging. Security controls complete.
Week 3: UI/UX polish + Slack/Teams integration + user training. Deployed to the pilot group.
The measure of success: employees in the pilot group reduce their ChatGPT usage because the internal tool gives better answers to their actual work questions. Voluntary adoption is the only adoption that lasts.
Sunil — CEO, Ailoitte. We build secure internal AI assistants for regulated industries: 3–4 week deployment, SSO integration, department-scoped access, full audit logging. ailoitte.com
Top comments (0)