Security for production LLM systems requires defense in depth. Unlike traditional APIs, large language models accept arbitrary natural language input and emit nondeterministic output, which expands the attack surface beyond standard web application vulnerabilities. This guide covers practical, code-first patterns for building secure LLM applications, from input sanitization to output validation, and explains how your inference provider shapes the security economics of your stack.
1. Map Your LLM Threat Model
Before writing middleware, identify what you are defending against. The OWASP Top 10 for LLM Applications highlights prompt injection, insecure output handling, sensitive information disclosure, and excessive agency. In practice, most production incidents fall into three categories: malicious input that manipulates system behavior, accidental output of secrets or toxic content, and infrastructure abuse such as model theft or prompt stuffing. Your controls should address all three layers.
2. Input Sanitization and Prompt Firewalling
Never forward raw user input directly to a model. Treat prompts like user-generated content that must pass a firewall. Implement a validation layer that checks for jailbreak heuristics, restricts prompt length, and strips or redacts sensitive entities.
A lightweight Python guardrail can run synchronously before any call to Oxlo.ai:
import os, re
from openai import OpenAI
client = OpenAI(
base_url="https://api.oxlo.ai/v1",
api_key=os.environ["OXLO_API_KEY"]
)
HEURISTICS = [
re.compile(r"ignore previous instructions", re.IGNORECASE),
re.compile(r"system prompt", re.IGNORECASE),
re.compile(r"</?script>", re.IGNORECASE),
]
def sanitize(text: str) -> str:
if len(text) > 8000:
raise ValueError("Input exceeds length threshold.")
for h in HEURISTICS:
if h.search(text):
raise ValueError("Input blocked by security heuristic.")
# Redact SSN-like patterns
text = re.sub(r"\b\d{3}-\d{2}-\d{4}\b", "[SSN_REDACTED]", text)
return text
def chat(user_input: str):
clean = sanitize(user_input)
return client.chat.completions.create(
model="your-model-id", # e.g., Llama 3.3 70B or Qwen 3 32B
messages=[{"role": "user", "content": clean}],
max_tokens=512,
)
For higher-risk workloads, add a secondary classification model in the loop. Because Oxlo.ai offers flat per-request pricing, running an additional moderation or classification call on every user interaction does not scale in cost with token count. That predictability makes multi-model guardrails economically viable. See https://oxlo.ai/pricing for current plans.
3. Output Validation and Structured Generation
Raw free-text output is hard to validate. Whenever possible, constrain the model with JSON mode or function calling so you can apply schema validation before acting on the result. If the model emits tool arguments, validate them against an allowlist before execution.
import jsonschema
from pydantic import BaseModel
class SearchQuery(BaseModel):
query: str
max_results: int
SCHEMA = SearchQuery.model_json_schema()
def safe_chat(user_input: str):
clean = sanitize(user_input)
resp = client.chat.completions.create(
model="your-model-id", # e.g., Qwen 3 32B
messages=[{"role": "user", "content": clean}],
response_format={"type": "json_object"},
)
raw = resp.choices[0].message.content
try:
jsonschema.validate(instance=json.loads(raw), schema=SCHEMA)
except jsonschema.ValidationError as e:
raise ValueError(f"Output schema violation: {e}")
return raw
Oxlo.ai supports JSON mode, streaming, and function calling across its LLM catalog, so you can enforce these constraints without switching providers or rewriting client code.
4. API Key Hygiene and Scoped Access
Store inference API keys in a secrets manager, never in client-side code or version control. Rotate keys quarterly, and use separate keys per environment. If your application exposes LLM features to end users, proxy all requests through your backend so the Oxlo.ai key remains server-side. For multi-tenant systems, maintain distinct key contexts per tenant so a revoked key limits blast radius to a single customer.
5. Rate Limiting and Abuse Economics
Rate limiting stops brute-force prompt injection and prevents runaway agents from draining budgets. Under token-based pricing, an attacker can artificially inflate costs by padding prompts or forcing verbose completions. Oxlo.ai uses request-based pricing, which means each API call costs one flat amount regardless of input length. This design removes the economic incentive behind prompt-stuffing attacks and makes your security scanning layer, such as secondary moderation requests, cost-predictable even at high volume.
6. PII Redaction and Data Privacy
Assume every prompt could contain personally identifiable information. Redact PII before it leaves your network, and avoid logging raw prompts to plaintext files. If you use retrieval-augmented generation, ensure your vector store enforces the same access controls as your primary database. When evaluating inference providers, confirm data handling policies align with your compliance requirements. Oxlo.ai processes requests through an API compatible with standard OpenAI SDK patterns, so you can insert your own privacy middleware without architectural changes.
7. Auditing and Observability
Log structured metadata for every inference request: timestamp, model, caller identity, latency, and a hashed fingerprint of the prompt. Avoid storing full prompts and outputs in centralized logs unless encrypted. Use these logs to detect anomalies, such as a single caller rapidly cycling through jailbreak templates. Oxlo.ai's flat request pricing simplifies cost attribution, making it easier to correlate budget spikes with security events.
8. Infrastructure and Endpoint Security
Your inference endpoint is a critical dependency. Cold starts add latency that can break circuit breakers and complicate timeout tuning. Oxlo.ai serves popular models with no cold starts, which means security middleware, retries, and circuit breakers behave consistently. The platform supports models across seven categories, including LLMs, code, vision, audio, embeddings, and image generation, all through the same OpenAI-compatible base URL. That uniformity lets you enforce security policies once and apply them across every model type your application consumes.
9. Putting It Together
The following FastAPI route demonstrates a consolidated pattern: sanitize input, call an Oxlo.ai model with JSON mode, validate the output, and return a safe response.
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
import json, jsonschema, os, re
from openai import OpenAI
app = FastAPI()
client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key=os.environ["OXLO_API_KEY"])
class UserRequest(BaseModel):
message: str
class SafeOutput(BaseModel):
answer: str
confidence: float
OUTPUT_SCHEMA = SafeOutput.model_json_schema()
HEURISTICS = [re.compile(r"ignore previous instructions", re.IGNORECASE)]
@app.post("/ask")
def ask(req: UserRequest):
# Input guard
if len(req.message) > 4000:
raise HTTPException(status_code=400, detail="Input too long.")
for h in HEURISTICS:
if h.search(req.message):
raise HTTPException(status_code=400, detail="Input rejected.")
# Inference
try:
r = client.chat.completions.create(
model="your-model-id", # e.g., DeepSeek R1 671B MoE
messages=[{"role": "user", "content": req.message}],
response_format={"type": "json_object"},
max_tokens=1024,
)
text = r.choices[0].message.content
except Exception:
raise HTTPException(status_code=502, detail="Inference failed.")
# Output guard
try:
data = json.loads(text)
jsonschema.validate(instance=data, schema=OUTPUT_SCHEMA)
except Exception:
raise HTTPException(status_code=500, detail="Output validation failed.")
return data
This pattern keeps secrets server-side, validates both directions of the data flow, and relies on Oxlo.ai's OpenAI-compatible endpoint so you can swap models without touching the security layer.
Conclusion
Building secure LLM systems is not a single feature. It is a stack of input guards, output validators, access controls, and infrastructure choices. Oxlo.ai fits into this stack as a predictable, compatible inference layer. Its request-based pricing removes the cost unpredictability that often discourages thorough input scanning, and its OpenAI SDK compatibility lets you drop it into existing security pipelines without rewriting client code. For teams shipping agentic or long-context workloads, that combination of cost control and operational consistency makes Oxlo.ai a strong foundation for secure AI infrastructure.
Top comments (0)