Implementing Privacy by Default for LLMs

LLM deployments often treat privacy as an afterthought. Prompts travel across third-party APIs, conversation history accumulates in databases, and sensitive data embeds itself into model weights through fine-tuning or feedback loops. Privacy by default inverts this model. It treats every token as potentially sensitive unless proven otherwise, and it bakes data minimization into the architecture rather than the terms of service. For engineering teams, this means building inference pipelines where PII never reaches the model provider, context windows are ephemeral, and compliance is a structural guarantee, not a checkbox.

The Privacy Problem in LLM Inference

Most managed LLM APIs log prompts for debugging, retain them for abuse monitoring, and may use them to improve models. Even when providers offer enterprise privacy terms, the data still leaves your network. For applications handling medical records, financial data, or proprietary source code, this exposure violates GDPR, HIPAA, and CCPA principles by default. The only robust fix is to assume the provider will see the prompt, and to ensure what arrives contains zero sensitive identifiers.

Core Principles of Privacy by Default

• Data minimization. Send only the data the model needs to fulfill the request. Strip metadata, system logs, and user identifiers.
• Purpose limitation. Use the model for a single, declared task. Do not allow secondary processing or training on your inputs.
• Storage limitation. Retain neither prompts nor completions on your servers or the provider's. Use transient context windows.
• Transparency. Log only hashes or tokens, never plaintext PII. Make your data flow auditable.

Architectural Patterns for Private Inference

Three patterns dominate production deployments that take privacy seriously.

Privacy Gateway. Deploy a middleware layer between your application and the inference API. The gateway scrubs PII, enforces rate limits, and inspects responses for data leaks before they reach the client.

Client-Side Tokenization. Map sensitive entities to opaque tokens before the prompt leaves your VPC. A reverse mapping table stays inside your infrastructure. The LLM operates on pseudonymous text.

Ephemeral Sessions. Avoid server-side conversation history. Send the full context required for multi-turn logic inside each request, or store history client-side encrypted with keys the provider never holds.

Implementing a Privacy Gateway

The following FastAPI service demonstrates a minimal privacy gateway. It scrubs email addresses and Social Security numbers from user messages, forwards the sanitized payload to an inference endpoint, and returns the completion without persisting either side of the conversation.

import os

import re

from fastapi import FastAPI

from pydantic import BaseModel

from openai import AsyncOpenAI

app = FastAPI()

client = AsyncOpenAI(

    base_url="https://api.oxlo.ai/v1",

    api_key=os.environ["OXLO_API_KEY"]

)

class ChatRequest(BaseModel):

    messages: list

    model: str = "llama-3.3-70b"

PII_PATTERNS = {

    "email": r"[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+.[