Building a Secure Local AI Agent on Open-Source Infrastructure: Lessons from SGLang, Olares, and Real-World Audits

Originally published on tamiz.pro.

The promise of Local AI is data sovereignty. By running Large Language Models (LLMs) entirely on-premise, organizations eliminate the risk of proprietary data leaking to third-party APIs. However, "local" does not automatically mean "secure." In fact, a poorly configured local LLM stack can expose your infrastructure to sophisticated attack vectors, including prompt injection, tool misuse, and adversarial inference attacks.

This article provides a technical analysis of building a secure local AI agent architecture using SGLang (a high-performance LLM serving engine) and Olares (a framework for local AI orchestration and security). We will dissect the security implications of these components, review lessons from real-world penetration testing, and provide a production-hardened deployment pattern.

The Security Landscape of Local LLMs

Before diving into the stack, we must redefine the threat model. Unlike traditional server-side applications, LLMs introduce unique vulnerabilities:

1. Prompt Injection: Malicious inputs designed to override system instructions.
2. Tool Use Exploitation: If an LLM has access to APIs (e.g., database queries, file systems), an attacker can trick it into executing arbitrary commands.
3. Data Leakage via Context: Sensitive information stored in the context window can be extracted via adversarial queries.
4. Model Poisoning: If you fine-tune models locally, the training data integrity is paramount.

Most organizations assume that because the model runs locally, it is safe. This is a fallacy. The boundary of trust has shifted from the network perimeter to the prompt interface.

Component Analysis: SGLang

What is SGLang?

SGLang (Structured Generation Language) is an open-source LLM serving engine developed by the Princeton University Data System Group. It is designed for high-throughput serving and complex structured generation (like JSON, regex, and programmatic outputs).

Security Implications

SGLang’s architecture offers several security advantages but also introduces specific risks:

• Structured Output Enforcement: SGLang allows you to define strict schemas for LLM outputs. This is a critical security control. By forcing the LLM to output valid JSON or adhere to a specific regex pattern, you reduce the surface area for prompt injection and data leakage. If the output doesn’t match the schema, the request can be rejected before it reaches downstream systems.
• RadixAttention: This optimization caches prefixes of similar prompts. While it improves performance, it can inadvertently cache sensitive context. Ensure that your context window management is rigorous to prevent leakage of previous sensitive interactions.
• Custom Backend Support: SGLang supports custom backends, allowing you to integrate with private VPCs. However, this complexity increases the attack surface if not configured correctly.

Code Example: Secure Structured Output with SGLang

import sglang as sgl
import json

# Define a structured output schema for a financial analyst agent
@sgl.function
def financial_analysis(s, query):
    s += sgl.user("Analyze the following financial data: " + query)
    s += sgl.assistant(
        sgl.gen("analysis", 
                stop=["</analysis>"],
                max_new_tokens=500,
                # Enforce JSON structure to prevent arbitrary text injection
                regex=r"\{\s*\"summary\":\s*\"[^\"]*\"\s*\}\"
               )
    )

# Initialize the engine
engine = sgl.Engine(model_path="meta-llama/Meta-Llama-3-8B-Instruct")

# Run the function
state = engine.run(financial_analysis, query="Revenue increased by 20%.")

# Validate and parse output
try:
    result = json.loads(state.text())
    print("Secure Output:", result)
except json.JSONDecodeError:
    print("Rejected: Output did not match schema")

Component Analysis: Olares

What is Olares?

Olares is an emerging framework focused on local AI orchestration with a strong emphasis on security and privacy. It provides tools for managing AI agents, handling tool execution, and enforcing security policies within a local environment.

Security Implications

Olares addresses several key security challenges:

• Sandboxed Tool Execution: Olares allows you to define tools that run in isolated environments. This prevents an LLM from executing arbitrary code on your host machine.
• Policy Enforcement: You can define policies that restrict which tools an agent can access based on user roles or data sensitivity.
• Audit Logging: Olares provides detailed logs of agent actions, which is crucial for forensic analysis in case of a security incident.

Architecture: The Olares Security Layer

Olares acts as a middleware between the LLM serving engine (like SGLang) and the external tools. It intercepts the LLM’s tool calls and validates them against a security policy before execution.

# Pseudocode for Olares security policy enforcement
from olares import Agent, PolicyEngine

policy = PolicyEngine(rules=[
    "DENY file_system_write",
    "ALLOW database_read",
    "ALLOW internet_request if domain in whitelisted_domains"
])

agent = Agent(
    model_path="local-llama-3-8b",
    tools=["db_query", "web_search"],
    policy_engine=policy
)

# This call would be blocked by the policy engine
agent.run("Write 'malicious' to /etc/passwd")

Real-World Audits: Lessons Learned

We have conducted internal and external security audits on local AI agent deployments. Here are the key findings:

1. The "Context Window" Leak

Finding: Many developers store sensitive PII (Personally Identifiable Information) in the context window to provide the LLM with relevant data. However, this data remains in memory and can be extracted via adversarial prompts.

Mitigation: Use retrieval-augmented generation (RAG) with strict access controls. Never store raw PII in the context window. If you must, use differential privacy techniques or synthetic data.

2. Tool Use Hallucination

Finding: LLMs can hallucinate tool inputs. An attacker can exploit this by crafting prompts that cause the LLM to call tools with unintended arguments.

Mitigation: Always validate tool inputs against a strict schema. Use SGLang’s structured output features to enforce valid JSON structures for tool calls. Implement a "human-in-the-loop" approval step for high-risk actions.

3. Model Inversion Attacks

Finding: Even with local models, adversaries can perform inference attacks to extract training data if the model is overfitted or if the API is exposed without rate limiting.

Mitigation: Implement rate limiting and anomaly detection. Regularly audit model outputs for signs of data leakage. Consider using model watermarking techniques.

Production-Ready Deployment Pattern

To build a secure local AI agent, we recommend the following architecture:

1. Isolation: Run the LLM serving engine (SGLang) and the orchestration layer (Olares) in separate containers.
2. Network Segmentation: Ensure the LLM service is not directly exposed to the internet. Use a reverse proxy (e.g., Nginx or Traefik) with mTLS for internal communication.
3. Policy Enforcement: Use Olares to enforce security policies before tool execution.
4. Audit Logging: Log all prompts, responses, and tool calls to a secure, immutable log storage system.
5. Monitoring: Monitor for anomalous patterns in token usage and prompt content.

Docker Compose Configuration

version: '3.8'
services:
  sglang-engine:
    image: lmsysorg/sglang:latest
    ports:
      - "30000:30000"
    environment:
      - MODEL_PATH=/models/meta-llama-3-8b
    volumes:
      - ./models:/models
    networks:
      - ai-network

  olares-agent:
    image: olares/agent:latest
    ports:
      - "8000:8000"
    environment:
      - SGLANG_ENDPOINT=http://sglang-engine:30000
      - POLICY_FILE=/etc/olares/policy.json
    volumes:
      - ./policy.json:/etc/olares/policy.json
    networks:
      - ai-network

networks:
  ai-network:
    driver: bridge

Best Practices for Secure Local AI

• Regular Updates: Keep SGLang and Olares updated to patch known vulnerabilities.
• Input Sanitization: Sanitize all user inputs before passing them to the LLM.
• Output Validation: Validate all LLM outputs against a strict schema before using them in downstream systems.
• Access Control: Implement strict access controls for who can interact with the AI agent.
• Data Minimization: Only provide the LLM with the minimum necessary data to perform its task.

Conclusion

Building a secure local AI agent is not just about choosing the right model; it’s about designing a secure architecture around it. By leveraging SGLang for structured, high-performance serving and Olares for security-conscious orchestration, you can build a robust local AI system. However, vigilance is key. Regular audits, strict policy enforcement, and a deep understanding of LLM-specific vulnerabilities are essential for maintaining security in production.

Frequently Asked Questions

Q: Is SGLang faster than vLLM for local deployment?
A: SGLang is often faster for structured generation and complex routing tasks due to its RadixAttention and structured output capabilities. For simple text generation, vLLM may be competitive. Benchmark both for your specific use case.

Q: How do I prevent prompt injection in a local LLM?
A: Use structured output enforcement (e.g., JSON schemas) to limit the LLM’s output format. Implement input sanitization and use a policy engine like Olares to validate tool calls.

Q: Can I use Olares with models other than Llama?
A: Yes, Olares is model-agnostic. It can work with any LLM served via a compatible interface, including SGLang, vLLM, or Ollama.