Building a Privacy-First, Zero-Cost AI Proxy for Public Data with Open-Source Tooling

Originally published on tamiz.pro.

In the current era of Large Language Models (LLMs), the tension between utility and privacy is palpable. While public data—such as documentation, public forums, or open datasets—is technically 'public,' sending it through a commercial AI API often involves leaking metadata, user patterns, and proprietary prompt engineering to a third-party provider. For engineers building production systems, the goal is to decouple the application logic from the AI provider while ensuring that no sensitive identifiers leak into the model's training set or the provider's logs.

This guide demonstrates how to build a production-ready, privacy-first AI proxy. We will use LiteLLM as the orchestration layer, Nginx as the secure gateway, and leverage the "Free Tier" ecosystem (Oracle Cloud or Google Cloud Free Tier) to keep the operational cost at exactly $0. By implementing a proxy layer, you gain the ability to implement PII (Personally Identifiable Information) scrubbing, request logging, and provider failover without changing a single line of code in your client application.

• 1. Architecture Overview
• 2. The Zero-Cost Infrastructure Stack
• 3. Step-by-Step Implementation
  • 3.1 Server Hardening and Prerequisites
  • 3.2 Deploying LiteLLM via Docker
  • 3.3 Configuring the Privacy Layer (PII Masking)
  • 3.4 Setting up Nginx as a Reverse Proxy with SSL
• 4. Implementing Privacy Filters and Guardrails
• 5. Testing the Proxy and Validating Anonymization
• 6. Production Considerations and Scaling
• 7. Frequently Asked Questions

1. Architecture Overview

The system follows a "Gateway Pattern." Instead of your application calling OpenAI, Anthropic, or Mistral directly, it calls your self-hosted proxy. The proxy intercepts the request, scrubs sensitive data, manages the API keys, and routes the request to the cheapest or most performant available model.

Data Flow:
Client App $\rightarrow$ Nginx (TLS Termination) $\rightarrow$ LiteLLM (Routing/Masking) $\rightarrow$ LLM Provider API $\rightarrow$ LiteLLM $\rightarrow$ Nginx $\rightarrow$ Client App

Why this approach?

1. Key Isolation: Your application never sees the actual provider API keys. If a client app is compromised, you rotate one proxy key, not ten provider keys.
2. Provider Agnostic: You can switch from GPT-4 to Claude 3 or a local Llama 3 instance by changing a config file, not your codebase.
3. Privacy Scrubbing: By inserting a masking layer, you ensure that even if public data contains an accidental email address or IP, it is replaced with a placeholder (e.g., [EMAIL_1]) before leaving your server.

2. The Zero-Cost Infrastructure Stack

To achieve a $0 monthly bill, we utilize the generous free tiers of cloud providers and the efficiency of lightweight open-source binaries.

Component	Tool	Why?	Cost
Compute	Oracle Cloud (Always Free)	4 ARM Ampere A1 cores, 24GB RAM	$0
Proxy Engine	LiteLLM	OpenAI-compatible API for 100+ LLMs	$0
Web Server	Nginx	High-performance load balancing & SSL	$0
SSL/TLS	Let's Encrypt	Industry standard automated certificates	$0
Containerization	Docker	Environment parity and easy updates	$0

3. Step-by-Step Implementation

3.1 Server Hardening and Prerequisites

Assuming you have a fresh Ubuntu 22.04 LTS instance on your free VPS, start by securing the perimeter.

# Update system packages
sudo apt update && sudo apt upgrade -y

# Install essential dependencies
sudo apt install -y curl git ufw docker.io docker-compose

# Configure Firewall to allow only HTTP, HTTPS, and SSH
sudo ufw allow OpenSSH
sudo ufw allow 'Nginx Full'
sudo ufw --force enable

# Ensure Docker starts on boot
sudo systemctl enable docker
sudo systemctl start docker

3.2 Deploying LiteLLM via Docker

LiteLLM acts as the core engine. It translates a single OpenAI-style API format into the specific formats required by various providers.

Create a directory for your proxy and a docker-compose.yml file:

mkdir ai-proxy && cd ai-proxy
nano docker-compose.yml

Paste the following configuration:

version: '3.9'
services:
  litellm:
    image: ghcr.io/berriai/litellm:main-latest
    container_name: litellm
    ports:
      - "4000:4000"
    volumes:
      - ./litellm_config.yaml:/app/config.yaml
    environment:
      - OPENAI_API_KEY=sk-your-openai-key
      - ANTHROPIC_API_KEY=sk-your-anthropic-key
      - LITELLM_MASTER_KEY=sk-proxy-master-key-123
    command: ["--config", "/app/config.yaml"]
    restart: always

Now, create the litellm_config.yaml file. This is where we define our models and the privacy settings.

model_list:
  - model_name: gpt-4-privacy
    litellm_params:
      model: openai/gpt-4-turbo
      api_key: "os.environ/OPENAI_API_KEY"
  - model_name: claude-privacy
    litellm_params:
      model: anthropic/claude-3-opus-20240229
      api_key: "os.environ/ANTHROPIC_API_KEY"

litellm_settings:
  drop_params: True # Drops parameters not supported by the target LLM
  set_verbose: False

Start the proxy:

sudo docker-compose up -d

3.3 Configuring the Privacy Layer (PII Masking)

To make this "Privacy-First," we cannot simply pass the text through. We need to intercept the prompt. While LiteLLM handles the routing, for deep PII scrubbing, we can use a custom middleware or a sidecar container. However, a lightweight way to implement this without adding latency is using a Python-based preprocessing script or LiteLLM's built-in logging hooks to track what is being sent.

For a truly robust privacy-first approach, we will implement a simple Presidio (Microsoft's PII anonymizer) wrapper. Since we are on a $0 VPS, we'll run a lightweight Python script as a middleware proxy using FastAPI.

Create privacy_filter.py:

from fastapi import FastAPI, Request
import httpx
from presidio_analyzer import AnalyzerEngine
from presidio_anonymizer import AnonymizerEngine

app = FastAPI()
analyzer = AnalyzerEngine()
anonymizer = AnonymizerEngine()

LITELLM_URL = "http://localhost:4000"

async def scrub_text(text: str):
    results = analyzer.analyze(text=text, entities=[], language='en')
    anonymized_result = anonymizer.anonymize(text=text, analyzer_results=results)
    return anonymized_result.text

@app.post("/v1/chat/completions")
async def proxy_chat(request: Request):
    body = await request.json()
    # Scrub the messages content
    for message in body.get("messages", []):
        message["content"] = await scrub_text(message["content"])

    async with httpx.AsyncClient() as client:
        response = await client.post(f"{LITELLM_URL}/v1/chat/completions", json=body)
        return response.json()

if __name__ == "__main__":
    import uvicorn
    uvicorn.run(app, host="0.0.0.0", port=8080)

Update your docker-compose.yml to include this privacy filter and expose port 8080 instead of 4000.

3.4 Setting up Nginx as a Reverse Proxy with SSL

Exposing a Python app directly to the web is a security risk. We use Nginx for SSL termination and request limiting.

sudo apt install -y nginx certbot python3-certbot-nginx

Create a configuration file for your site:

sudo nano /etc/nginx/sites-available/ai-proxy

server {
    listen 80;
    server_name proxy.yourdomain.com;

    location / {
        proxy_pass http://localhost:8080;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

Enable the site and obtain an SSL certificate:

sudo ln -s /etc/nginx/sites-available/ai-proxy /etc/nginx/sites-enabled/
sudo nginx -t
sudo systemctl restart nginx

# Run Certbot for SSL (Follow prompts)
sudo certbot --nginx -d proxy.yourdomain.com

4. Implementing Privacy Filters and Guardrails

In a production environment, "privacy-first" means more than just removing emails. You need to consider Prompt Injection and Data Leakage.

Contextual Anonymization

Instead of just deleting data, use Pseudonymization. If a user mentions "John Doe" three times in a prompt, replace all instances with [PERSON_1]. This preserves the semantic relationship for the LLM while hiding the identity. This is handled by the AnonymizerEngine in the code provided in section 3.3.

Request Rate Limiting

To prevent your free VPS from being overwhelmed or your API credits from being drained by a bot, add rate limiting to your Nginx config:

# Add to nginx.conf http block
limit_req_zone $binary_remote_addr zone=ai_limit:10m rate=5r/s;

# Add to server block
location / {
    limit_req zone=ai_limit burst=10 nodelay;
    proxy_pass http://localhost:8080;
}

5. Testing the Proxy and Validating Anonymization

To verify that the privacy layer is working, send a request containing PII to your new proxy endpoint.

Request:

curl https://proxy.yourdomain.com/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer sk-proxy-master-key-123" \
  -d '{
    "model": "gpt-4-privacy",
    "messages": [{"role": "user", "content": "My email is [e-mail] and my phone is 555-0199. Can you summarize this?"}]
  }'

Verification:
Check the logs of the litellm container. You should see that the text arriving at LiteLLM is:
"My email is <EMAIL_ADDRESS> and my phone is <PHONE_NUMBER>. Can you summarize this?"

This confirms that the PII never left your VPS in its raw form.

6. Production Considerations and Scaling

While this setup is free and effective, scaling it requires a few architectural shifts:

1. Redis for Caching: To reduce costs further, implement a caching layer using Redis. If two users ask the same question about public data, return the cached response instead of hitting the API. LiteLLM has native support for Redis caching.
2. Health Checks: Use a tool like Tamiz's Insights or a simple Prometheus exporter to monitor the latency added by the PII scrubbing layer. The presidio-analyzer can be CPU-intensive.
3. Asynchronous Processing: For very large public datasets, avoid synchronous HTTP requests. Move to a task queue like Celery or RabbitMQ to process the data, scrub it, and then call the LLM.
4. Log Rotation: Nginx and Docker logs can quickly fill up a small VPS disk. Configure logrotate to keep only 7 days of logs.

7. Frequently Asked Questions

Q: Does this increase latency significantly?

A: Yes, adding a PII scrubbing layer adds roughly 50ms to 200ms of latency per request depending on the length of the text. However, for most AI applications, the LLM's generation time (seconds) far outweighs the proxy overhead (milliseconds).

Q: Can I use this for local models instead of APIs?

A: Absolutely. You can run Ollama on the same VPS (if you have the RAM) and point LiteLLM to the local Ollama endpoint. This would make the system 100% private as data never leaves your hardware.

Q: Is the $0 VPS powerful enough for Presidio?

A: Yes, on an Oracle Cloud ARM instance (4 cores/24GB RAM), Presidio runs comfortably. If you are on a smaller 1GB RAM instance, you may need to swap to a simpler regex-based scrubbing method to avoid Out-Of-Memory (OOM) kills.