Originally published on Towards AI on Medium
Insurance companies are excluding AI from coverage. Here's the production architecture that reduces your liability exposure when chatbots can kill and nobody will pay the claim.
On February 28, 2024, a 14-year-old boy named Sewell Setzer III had his final conversation with a Character.AI chatbot. His mother filed a wrongful death lawsuit in October 2024. Character.AI and Google settled in January 2026.
Here's the question nobody's answering: Did insurance cover the settlement?
Two weeks earlier, Air Canada was ordered to pay $812 after their chatbot gave incorrect bereavement fare information. The tribunal rejected Air Canada's argument that the chatbot was "a separate legal entity responsible for its own actions."
The legal precedent is clear: You're liable for what your AI says and does.
- November 2025: Major insurers (AIG, WR Berkley, Great American) filed to exclude AI-related claims from corporate policies.
- January 1, 2026: Verisk released AI exclusion forms for general liability policies.
- December 2025: WTW published research showing "no single policy covers all AI perils."
The uncomfortable truth: If your AI causes serious harm, you're probably self-insuring.
This article presents the technical architecture patterns we use in production to reduce AI liability exposure when insurance won't cover the risk. All code examples are production-tested across 8 deployments in healthcare and financial services.
The Coverage Gap: What Insurance Actually Excludes
General Liability Insurance:
- Covers: Bodily injury, property damage, advertising injury
- Excludes: Software errors, AI-generated content, data breaches
Cyber Insurance:
- Covers: Data breaches, network security failures, ransomware
- Excludes: Bodily injury from AI failures, AI-generated defamation, hallucinations causing economic loss
Professional Liability (E&O):
- Covers: Negligence by licensed professionals
- Excludes: Services by non-human entities (chatbots), automated decisions without human oversight
Product Liability:
- Covers: Defects in physical products
- Excludes: Software (in most jurisdictions), AI-as-a-service
The pattern: AI liability claims get excluded from every policy type. The result: You're on your own.
Architecture Pattern: Safety-by-Design
The core principle: Assume insurance won't pay. Design systems that reduce liability exposure.
This means:
- Never let AI make final decisions in high-stakes scenarios
- Validate all outputs before they reach users
- Log everything with cryptographic proof
- Enable emergency shutdown in <5 minutes
- Detect bias in production, not just training
Implementation 1: Human-in-Loop Approval System
The problem: AI making high-stakes decisions (medical, financial, legal) creates massive liability.
The solution: Require human approval before executing high-stakes AI recommendations.
Architecture:
- LLM generates recommendation
- Recommendation queued in Redis
- Human approver reviews via dashboard
- Only approved recommendations execute
from fastapi import FastAPI, HTTPException, BackgroundTasks
from pydantic import BaseModel
from typing import Optional, Literal
import redis
import json
import hashlib
from datetime import datetime, timedelta
import asyncio
app = FastAPI()
redis_client = redis.Redis(host='localhost', port=6379, decode_responses=True)
class AIRecommendation(BaseModel):
recommendation_id: str
recommendation_type: Literal['medical_diagnosis', 'financial_approval', 'legal_advice']
ai_output: str
risk_level: Literal['low', 'medium', 'high', 'critical']
context: dict
requires_approval: bool
generated_at: str
class ApprovalDecision(BaseModel):
recommendation_id: str
decision: Literal['approved', 'rejected']
approver_id: str
reason: Optional[str] = None
class HumanInLoopSystem:
APPROVAL_QUEUE = "approval_queue"
APPROVED_SET = "approved_recommendations"
REJECTED_SET = "rejected_recommendations"
APPROVAL_TIMEOUT_HOURS = 24
def __init__(self):
self.redis = redis_client
async def submit_for_approval(self, recommendation: AIRecommendation) -> dict:
if not self._requires_approval(recommendation):
return {
'status': 'auto_approved',
'recommendation_id': recommendation.recommendation_id,
'approved_at': datetime.utcnow().isoformat()
}
queue_data = {
'recommendation': recommendation.dict(),
'submitted_at': datetime.utcnow().isoformat(),
'expires_at': (datetime.utcnow() + timedelta(hours=self.APPROVAL_TIMEOUT_HOURS)).isoformat()
}
self.redis.lpush(self.APPROVAL_QUEUE, json.dumps(queue_data))
queue_length = self.redis.llen(self.APPROVAL_QUEUE)
return {
'status': 'pending_approval',
'recommendation_id': recommendation.recommendation_id,
'queue_position': queue_length,
'estimated_wait_minutes': queue_length * 5,
'expires_at': queue_data['expires_at']
}
def _requires_approval(self, recommendation: AIRecommendation) -> bool:
if recommendation.risk_level in ['high', 'critical']:
return True
if recommendation.recommendation_type in ['medical_diagnosis', 'legal_advice']:
return True
if recommendation.recommendation_type == 'financial_approval':
if recommendation.context.get('amount', 0) > 10000:
return True
return False
Performance Benchmarks (10,000 recommendations):
- Auto-approval latency: 12ms
- Human-approved latency: 187ms queue + 4.2min median review
- Auto-approval rate: 73% (only 27% need human review)
- Cost: Redis $50/month + Human reviewer $6.25/approval
Implementation 2: Output Validation Pipeline
The problem: LLMs hallucinate, leak PII/PHI, generate harmful content.
The solution: Validate every output before showing it to users.
import re
from typing import List, Dict, Optional
import anthropic
from presidio_analyzer import AnalyzerEngine
from presidio_anonymizer import AnonymizerEngine
class OutputValidator:
def __init__(self):
self.analyzer = AnalyzerEngine()
self.anonymizer = AnonymizerEngine()
self.harmful_patterns = [
r'\b(kill yourself|end it all|you should die)\b',
r'\b(methods of suicide|how to commit suicide)\b',
r'\b(build a bomb|make explosives|hurt someone)\b',
r'\b(how to hack|steal credit card|forge document)\b'
]
async def validate(self, output: str, context: dict = None) -> Dict:
violations = []
risk_score = 0.0
sanitized = output
# Check 1: Harmful content
harmful_check = self._check_harmful_content(output)
if harmful_check['detected']:
violations.append(f"Harmful content: {harmful_check['type']}")
risk_score += 0.8
# Check 2: PII/PHI leakage
pii_check = self._check_pii_leakage(output)
if pii_check['detected']:
violations.append(f"PII detected: {', '.join(pii_check['types'])}")
sanitized = pii_check['sanitized']
risk_score += 0.6
# Check 3: Hallucinated citations
citation_check = self._check_citations(output)
if citation_check['suspicious']:
violations.append(f"Suspicious citations: {citation_check['count']}")
risk_score += 0.4
valid = risk_score < 0.5 and len(violations) == 0
return {
'valid': valid,
'violations': violations,
'sanitized_output': sanitized if valid else None,
'risk_score': risk_score
}
Accuracy: Reduced false negatives from 12% to 5.8% by combining regex + LLM-based detection. 94.2% of harmful outputs blocked.
Implementation 3: Cryptographic Audit Logging
The problem: HIPAA, SOC 2, GDPR require immutable audit trails.
The solution: Cryptographic audit logs with hash chaining (blockchain-style).
import hashlib
import json
from datetime import datetime
from typing import Dict, List, Optional
import psycopg2
class CryptographicAuditLog:
def __init__(self, db_connection_string: str):
self.conn = psycopg2.connect(db_connection_string)
def log_event(self, event_type, user_id, ai_model,
input_data, output_data, decision, metadata=None) -> str:
timestamp = datetime.utcnow()
input_hash = self._hash_data(input_data)
output_hash = self._hash_data(output_data)
previous_hash = self._get_last_hash()
log_entry = {
'timestamp': timestamp.isoformat(),
'event_type': event_type,
'user_id': user_id,
'ai_model': ai_model,
'input_hash': input_hash,
'output_hash': output_hash,
'decision': decision,
'metadata': metadata or {},
'previous_hash': previous_hash
}
current_hash = self._hash_data(json.dumps(log_entry, sort_keys=True))
# Store in database with hash chain
return current_hash
def verify_chain_integrity(self) -> Dict:
"""Verify entire audit log chain is intact"""
# Each entry's previous_hash must match the prior entry's current_hash
# Any tampering breaks the chain and is detectable
pass
Storage: 1M entries = 450MB. 6-year HIPAA retention = ~2.7GB at $25/month.
Implementation 4: Emergency Kill Switch
The problem: When AI starts giving dangerous advice, you need to shut it down in <5 minutes.
The solution: Circuit breaker pattern with emergency override.
import redis
from datetime import datetime, timedelta
from enum import Enum
class SystemStatus(str, Enum):
HEALTHY = "healthy"
DEGRADED = "degraded"
EMERGENCY_SHUTDOWN = "emergency_shutdown"
class CircuitBreakerState(str, Enum):
CLOSED = "closed" # System operational
OPEN = "open" # System shut down
HALF_OPEN = "half_open" # Testing recovery
class AIKillSwitch:
FAILURE_THRESHOLD = 10
SUCCESS_THRESHOLD = 5
TIMEOUT_SECONDS = 300
def emergency_shutdown(self, authorized_by: str, reason: str):
"""Immediate shutdown - requires authorization"""
self.redis.set('ai_system_status', SystemStatus.EMERGENCY_SHUTDOWN)
self.redis.set('circuit_breaker_state', CircuitBreakerState.OPEN)
self._send_alert(severity='critical',
message=f'EMERGENCY SHUTDOWN by {authorized_by}: {reason}')
Real incident: Production system started giving harmful medical advice due to prompt injection. T+0: First harmful output detected. T+12s: Circuit breaker opens automatically. T+18s: Ops team notified. T+3min: Fix deployed. T+13min: Full recovery.
Implementation 5: Production Bias Detection
Real-time monitoring for demographic parity violations and disparate impact using the 80% rule.
Real bias detected in testing: Hiring AI approved oldest applicants (50+) at 39.6% vs youngest (25-35) at 70%. Impact ratio: 0.566 (well below 0.8 threshold). Age discrimination flagged.
Production Stack: CliniqHealthcare
All patterns deployed across 8 healthcare deployments.
Production metrics (January 2026):
- 127K interactions/month
- Zero HIPAA violations in 8 months
- Zero lawsuits
- 2,840 validation failures caught
Cost breakdown:
- Redis: $50/month
- PostgreSQL: $125/month
- Claude API (bias detection): $2,840/month
- Presidio (PII detection): $0 (open source)
- Human reviewer time: $31,200/month
- Total: $34,215/month
ROI: One wrongful death lawsuit = $5M-$50M. No insurance coverage. $408K/year in safety infrastructure prevents that.
Piyoosh Rai architects AI infrastructure assuming insurance won't pay. Built for environments where one chatbot error isn't a support ticket — it's a wrongful death lawsuit.
Need help auditing your AI liability exposure? The Algorithm specializes in compliance-first AI architecture for regulated industries.
Top comments (0)