From Lab to Production: Securing Local LLMs and AI Agents with Self-Hosted Infrastructure and GitOps Guardrails

Originally published on tamiz.pro.

You've fine-tuned your model. It runs on your GPU server. Prompt injections are a lab curiosity. But the moment you expose it to real users — with tools, memory, and agent loops — the attack surface explodes. Local doesn't mean secure. In this deep dive, I'll walk you through the architecture, guardrails, and operational practices that turn a fragile lab prototype into a production-hardened self-hosted LLM system protected by GitOps.

Why "Local" Doesn't Mean Secure

Self-hosted LLMs offer privacy, cost control, and compliance advantages. But they also inherit every risk of any production system — plus new ones unique to generative AI. A local model serving over HTTP is just an API server, and API servers get probed, exploited, and exfiltrated. AI agents compound this with tool access, persistent memory, and autonomous action loops.

The lab mindset treats these as features. Production treats them as threat vectors. Bridging that gap requires infrastructure that is auditable, version-controlled, and relentlessly automated — exactly what GitOps delivers.

Architecture Overview

A production self-hosted LLM stack has four logical layers, each with its own security responsibilities:

┌─────────────────────────────────────────────────┐
│             GitOps Control Plane                │
│  (ArgoCD / Flux + OPA/Gatekeeper + SealedSeals) │
├─────────────────────────────────────────────────┤
│           Model Serving Layer                   │
│  (vLLM / TGI / llama.cpp + GPU node pools)      │
├─────────────────────────────────────────────────┤
│         Agent & Tool Execution Layer            │
│  (Sandboxed containers + policy enforcement)     │
├─────────────────────────────────────────────────┤
│         Data & Memory Layer                     │
│  (Encrypted stores + RAG pipeline + audit logs)  │
└─────────────────────────────────────────────────┘

Every layer is defined as code. Every deployment flows through a pull-request gate. Every runtime decision is policy-enforced, not opinion-enforced.

1. The GitOps Control Plane

GitOps is the backbone. Instead of imperatively applying Kubernetes manifests or Helm charts, you declare desired state in a Git repository and a controller (ArgoCD or Flux) reconciles the cluster to match.

Core components:

• ArgoCD for application lifecycle management, with application sets for multi-model or multi-environment deployments
• OPA/Gatekeeper as the admission controller, enforcing policies like "no image from untrusted registries" or "GPU nodes must have nvidia.com/gpu requests validated"
• SealedSecrets or External Secrets Operator for managing sensitive values (API keys, model weights encryption keys) without committing plaintext to Git
• Kyverno as a complementary policy engine for more complex rule logic that OPA's Rego can't express comfortably

Your Git repo structure should mirror your environment topology:

infra/
  clusters/
    prod/
      argocd/apps.yaml
      kyverno/policies.yaml
      sealed-secrets/
  applications/
    llm-serving/
      values.yaml
      kustomization.yaml
    agent-runtime/
      values.yaml
      kustomization.yaml
    rag-pipeline/
      values.yaml
      kustomization.yaml
  policies/
    network-policy.yaml
    pod-security.yaml
    image-policy.yaml

This structure makes it trivial to audit: a single git log --oneline across the infra/ tree tells you exactly what changed, when, and by whom.

2. The Model Serving Layer

Your model server is the most exposed component. It handles inbound traffic, loads weights into GPU memory, and streams outputs. Compromise here means full model theft, data poisoning, or worst-case — uncontrolled inference that drains your cluster.

Hardening steps:

Network isolation. Never expose the inference endpoint directly to the internet. Use a service mesh (Istio or Linkerd) with mutual TLS between all in-cluster components, and an ingress controller with WAF rules that strip or reject malformed prompt payloads.

Resource limits as attack surface reduction. GPU nodes are expensive and targeted. Set strict limits and requests on every pod. Use Kubernetes LimitRanges and ResourceQuotas to prevent a single compromised pod from consuming the entire node pool.

Image supply chain. Sign your serving images with Cosign and verify signatures at admission via Gatekeeper. Combine with in-toto attestation for build provenance.

# Example: Gatekeeper constraint for image signature verification
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sRequiredAnnotations
metadata:
  name: require-cosign-signature
spec:
  match:
    kinds:
      - apiGroups: ["apps"]
        kinds: ["Deployment"]
  parameters:
    annotations:
      - "cosign.sigstore.dev/signed"

Model weight protection. Weights are intellectual property. Encrypt them at rest using a KMS-integrated CSI driver like secrets-store-csi-driver with AWS KMS, GCP KMS, or HashiCorp Vault. Stream weights into the container at runtime only — never persist them on node local storage.

3. The Agent & Tool Execution Layer

AI agents are the new attack surface frontier. Unlike static models, agents compose themselves at runtime — selecting tools, making API calls, writing files, executing commands. Each tool invocation is a potential privilege escalation path.

Sandboxing is non-negotiable. Every agent execution should run in an isolated container or Pod with:

• Read-only filesystems where possible
• No privileged containers (allowPrivilegeEscalation: false)
• Network policies restricting egress to only the tools' required endpoints
• Dropped capabilities — remove all Linux capabilities except what's strictly necessary

# Example: Strict PodSecurityProfile for agent containers
apiVersion: v1
kind: Pod
metadata:
  name: agent-execution
spec:
  securityContext:
    runAsNonRoot: true
    runAsUser: 65534
    fsGroup: 65534
  containers:
    - name: agent
      image: registry.internal/agent-runtime:v2.3.1
      securityContext:
        allowPrivilegeEscalation: false
        capabilities:
          drop: ["ALL"]
        readOnlyRootFilesystem: true
      volumeMounts:
        - name: tmp
          mountPath: /tmp
        - name: agent-workdir
          mountPath: /home/agent/work
          readOnly: false
  volumes:
    - name: tmp
      emptyDir:
        sizeLimit: 100Mi
    - name: agent-workdir
      emptyDir:
        sizeLimit: 500Mi

Tool policy enforcement. Define which tools each agent identity can access via a policy layer (not hardcoded). Use Open Policy Agent to evaluate tool requests before they reach the execution sandbox.

# Example: OPA policy restricting agent tool access
package agent.tool_policy

denying_tool[msg] {
    input.agent_id == "read_write_agent"
    input.tool == "execute_command"
    msg := "read_write_agent cannot execute arbitrary commands"
}

denying_tool[msg] {
    input.tool == "write_file"
    not startswith(input.file_path, "/home/agent/work")
    msg := sprintf("write_file denied: path %s is outside allowed directory", [input.file_path])
}

denying_tool[msg] {
    input.tool == "http_request"
    not startswith(input.url, "https://")
    msg := "http_request denied: only HTTPS URLs are permitted"
}

Guardrails for model output. Beyond tool policies, enforce guardrails on what the model itself produces. Use frameworks like NeMo Guardrails or Guardrails AI to detect and block:

• Prompt injection attempts
• PII leakage in outputs
• Dangerous or disallowed content
• Excessive tool use (budget enforcement)

4. The Data & Memory Layer

RAG pipelines and agent memory (persistent conversations, tool outputs, external data lookups) are high-value targets. A compromised vector store or embedding model gives attackers indirect access to your proprietary data.

Encryption everywhere. Encrypt data at rest (AES-256 via your cloud KMS or Vault) and in transit (TLS 1.3 minimum). Use field-level encryption for particularly sensitive documents before they enter your vector store.

Access controls per document. Don't rely on a single vector store credential. Implement row-level or document-level access control by tagging embeddings with metadata that maps to RBAC policies, then filtering at query time.

Audit logging. Every embedding operation, retrieval query, and memory read/write should be logged to an immutable audit trail. Ship these logs to a separate, access-restricted SIEM or logging backend — never store them in the same cluster.

GitOps Workflows for LLM Systems

The architecture above is only as strong as the workflow that maintains it. GitOps isn't just about declarative deployments — it's about creating a verifiable chain of custody for every component in your LLM stack.

The Pull Request Gate

Every change to your infrastructure or model artifacts goes through a PR. This isn't ceremony; it's your primary security control.

PR checklist for LLM infrastructure changes:

1. Diff review. At least one other engineer must approve the change. For model-related changes, include someone who understands the threat model of that specific component.
2. Policy validation. Run kyverno test or conftest against your manifests before merging. Catch violations early.
3. Image verification. Ensure the referenced image tag exists in your signed registry and has a valid Cosign signature.
4. Secrets audit. Confirm no plaintext secrets appear in the diff. If new secrets are needed, they go through the SealedSecrets or External Secrets flow.
5. Rollback plan. Document the exact steps to revert if the change fails in production.

Automated Reconciliation and Drift Detection

ArgoCD or Flux continuously reconciles your cluster state against the Git repository. But reconciliation alone isn't enough — you need visibility into drift.

Configure alerts for:

• Out-of-sync applications — a pod spec was modified directly on the cluster without going through Git
• Health degradation — readiness probes failing on the serving layer
• Policy violations — Kyverno or Gatekeeper flags detected
• Failed syncs — automated rollbacks triggered

Set up a daily compliance report that summarizes the state of your cluster relative to Git:

# ArgoCD Application for automated compliance scanning
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: compliance-scanner
spec:
  source:
    repoURL: https://git.internal/security-policies.git
    targetRevision: main
    path: scans/daily
  destination:
    server: https://kubernetes.default.svc
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

Versioning Model Artifacts

Your model weights are part of your infrastructure. Treat them like code.

• Store model versions in a versioned artifact registry (NGC for NVIDIA models, Hugging Face Hub with private repos, or a self-hosted registry)
• Tag each model with a semver and link it to the Git commit that produced it via training metadata
• Reference models by digest, not by mutable tag, in your deployment manifests
• When promoting a model from staging to production, update only the reference in Git — the GitOps controller handles the rollout

# Reference model by digest for immutability
apiVersion: apps/v1
kind: Deployment
metadata:
  name: llm-serving
spec:
  template:
    spec:
      containers:
        - name: inference
          image: nvcr.io/nim/meta/llama-3.1-70b-instruct@sha256:a1b2c3...
          env:
            - name: MODEL_PATH
              value: "/models/llama-3.1-70b-instruct"

Threat Model: What You're Protecting Against

Before diving deeper into mitigations, let's be explicit about the threats. A self-hosted LLM system faces a distinct set of attack vectors:

Threat Category	Attack Vector	Impact
Inference abuse	Resource exhaustion via long prompts, recursive tool loops	Denial of service, GPU cost spike
Prompt injection	Malicious content embedded in retrieved documents or user input	Unauthorized tool use, data exfiltration
Supply chain	Compromised model weights, poisoned training data, malicious containers	Integrity compromise, backdoored inference
Data leakage	Unencrypted RAG retrievals, noisy side-channel outputs	PII exposure, IP theft
Agent hijacking	Manipulating agent memory or tool configurations	Persistent unauthorized access
Node escape	Exploiting container runtime vulnerabilities	Full cluster compromise
Credential theft	Intercepting secrets from pod environments or etcd	Lateral movement

Each of these maps to a specific defense layer in your architecture.

Defense-in-Depth: Layering Your Security Controls

No single control is sufficient. Here's how to stack them effectively.

Layer 1: Prevention at the Network Boundary

• WAF rules that detect and block common injection patterns before they reach your model
• Rate limiting per client identity to prevent resource exhaustion
• Request size limits — cap prompt length and tool call complexity

# Nginx ingress rate limiting configuration
limit_req_zone $binary_remote_addr zone=llm_api:10m rate=10r/s;

location /v1/chat/completions {
    limit_req zone=llm_api burst=20 nodelay;
    proxy_pass http://llm-serving:8000;
    proxy_set_header X-Request-ID $request_id;
}

Layer 2: Admission Control

Gatekeeper and Kyverno policies that reject any deployment violating your security baseline:

• No hostNetwork: true
• No privileged: true
• Images must come from your trusted registry
• All pods must have resource limits
• Service accounts must not be default

Layer 3: Runtime Security

Deploy a runtime security agent like Falco or Tetragon to detect anomalous behavior:

• Unexpected process execution inside the inference container
• Network connections to unrecognized destinations
• File system writes outside allowed paths
• Credential access patterns that deviate from baseline

# Falco rule: detect unexpected process in inference container
rule: Suspicious process in LLM container
desc: Detect if an unexpected process spawns in the inference pod
condition: >
  spawned_process and
  container.id != "host" and
  container.image starts with "nvcr.io/nim/" and
  not proc.name in (llama_server, vllm, triton, python, sh)
output: >
  "Suspicious process in inference container"
    "container=%container.name image=%container.image"
    "process=%proc.name pid=%proc.pid parent=%proc.pname"
priority: WARNING

Layer 4: Observability and Response

• Centralized logging with structured fields for every inference request (timestamp, client ID, model, token count, tool calls, latency)
• Trace propagation through the entire agent loop (OpenTelemetry)
• Automated incident response playbooks triggered on Falco/Tetragon alerts
• Regular red-team exercises targeting your agent tool chains

Practical Deployment: A Worked Example

Let's walk through deploying a complete self-hosted LLM system with GitOps guardrails. We'll use vLLM for serving, an agentic framework with sandboxed execution, and ArgoCD for GitOps management.

Step 1: Clone and Structure the Repository

mkdir -p llm-production/infra/clusters/prod
mkdir -p llm-production/infra/applications/serving
mkdir -p llm-production/infra/applications/agent-runtime
mkdir -p llm-production/infra/policies
mkdir -p llm-production/models/registry

Step 2: Define the Serving Layer

Create the vLLM deployment with hardened security context and resource limits:

# infra/applications/serving/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-serving
  namespace: llm-system
  labels:
    app: vllm-serving
    version: "1.0.0"
spec:
  replicas: 2
  selector:
    matchLabels:
      app: vllm-serving
  template:
    metadata:
      labels:
        app: vllm-serving
        version: "1.0.0"
    spec:
      automountServiceAccountToken: false
      securityContext:
        runAsNonRoot: true
        runAsUser: 1000
        fsGroup: 1000
      containers:
        - name: vllm
          image: vllm/vllm-openai:latest@sha256:<verified-digest>
          args:
            - serve
            - /models/llama-3.1-70b-instruct
            - --host
            - 0.0.0.0
            - --port
            - "8000"
            - --max-model-len
            - "32768"
          env:
            - name: VLLM_USE_PREFETCH
              value: "1"
            - name: HF_HOME
              value: "/mnt/models-cache"
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop: ["ALL"]
            seccompProfile:
              type: RuntimeDefault
          resources:
            requests:
              cpu: "8"
              memory: "32Gi"
              nvidia.com/gpu: "1"
            limits:
              cpu: "8"
              memory: "32Gi"
              nvidia.com/gpu: "1"
          volumeMounts:
            - name: model-storage
              mountPath: /mnt/models-cache
            - name: tmp
              mountPath: /tmp
          livenessProbe:
            httpGet:
              path: /health
              port: 8000
            initialDelaySeconds: 60
            periodSeconds: 15
          readinessProbe:
            httpGet:
              path: /health
              port: 8000
            initialDelaySeconds: 30
            periodSeconds: 10
      volumes:
        - name: model-storage
          persistentVolumeClaim:
            claimName: model-pvc
        - name: tmp
          emptyDir:
            sizeLimit: 1Gi

Step 3: Define the Agent Runtime with Sandboxing

# infra/applications/agent-runtime/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: agent-runtime
  namespace: llm-system
spec:
  replicas: 3
  selector:
    matchLabels:
      app: agent-runtime
  template:
    metadata:
      labels:
        app: agent-runtime
    spec:
      automountServiceAccountToken: false
      securityContext:
        runAsNonRoot: true
        runAsUser: 65534
        fsGroup: 65534
      containers:
        - name: agent
          image: registry.internal/agent-runtime:v2.3.1@sha256:<verified-digest>
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop: ["ALL"]
            readOnlyRootFilesystem: true
            seccompProfile:
              type: RuntimeDefault
          resources:
            requests:
              cpu: "2"
              memory: "4Gi"
            limits:
              cpu: "4"
              memory: "8Gi"
          env:
            - name: TOOL_EXECUTION_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            - name: AGENT_LOG_LEVEL
              value: "info"
          volumeMounts:
            - name: agent-workdir
              mountPath: /home/agent/work
            - name: tmp
              mountPath: /tmp
      volumes:
        - name: agent-workdir
          emptyDir:
            sizeLimit: 500Mi
        - name: tmp
          emptyDir:
            sizeLimit: 100Mi
---
# NetworkPolicy: restrict agent egress to only required services
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: agent-egress-policy
  namespace: llm-system
spec:
  podSelector:
    matchLabels:
      app: agent-runtime
  policyTypes:
    - Egress
  egress:
    - to:
        - podSelector:
            matchLabels:
              app: vllm-serving
      ports:
        - protocol: TCP
          port: 8000
    - to:
        - ipBlock:
            cidr: 10.0.0.0/8
      ports:
        - protocol: TCP
          port: 443
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
      ports:
        - protocol: UDP
          port: 53
        - protocol: TCP
          port: 53

Step 4: Define Kyverno Policies

# infra/policies/pod-security.yaml
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: enforce-pod-security
spec:
  validationFailureAction: Enforce
  background: true
  rules:
    - name: deny-privileged
      match:
        any:
          - resources:
              kinds: ["Pod"]
      validate:
        message: "Privileged containers are not allowed."
        pattern:
          spec:
            containers:
              - securityContext:
                  privileged: "false"
    - name: require-run-as-non-root
      match:
        any:
          - resources:
              kinds: ["Pod"]
      validate:
        message: "Pods must run as non-root."
        pattern:
          spec:
            securityContext:
              runAsNonRoot: true
    - name: require-resource-limits
      match:
        any:
          - resources:
              kinds: ["Pod"]
      validate:
        message: "All containers must have resource limits."
        pattern:
          containers:
            - resources:
                limits:
                  memory: "?"

Step 5: Configure ArgoCD for Auto-Sync

# infra/clusters/prod/argocd/apps.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: llm-system
  namespace: argocd
spec:
  project: llm-production
  source:
    repoURL: https://git.internal/llm-production-infra.git
    targetRevision: main
    path: infra/applications
  destination:
    server: https://kubernetes.default.svc
    namespace: llm-system
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
      - PruneLast=true
    retry:
      limit: 5
      backoff:
        duration: 30s
        factor: 2
        maxDuration: 5m

Maintenance: Keeping Production Secure Over Time

Deployment is the easy part. Maintaining security posture is the hard part.

Regular Audit Cadence

• Weekly: Review ArgoCD sync health and policy violation reports
• Bi-weekly: Rotate any secrets managed through SealedSecrets
• Monthly: Scan all container images for new CVEs; update base images
• Quarterly: Red-team exercise targeting the agent tool execution paths
• Annually: Full architecture review and threat model update

Incident Response Playbook

When an alert fires, have an automated and manual response ready:

1. Automated containment: NetworkPolicy update to isolate the affected pod
2. Log preservation: Snapshot of relevant logs and pod exec history before any cleanup
3. Manual investigation: Root cause analysis focusing on the weakest link
4. GitOps rollback: Revert to the last known-good Git commit if the incident was configuration-driven
5. Post-mortem: Document the failure mode and add or strengthen a policy to prevent recurrence

Model Drift and Poisoning Detection

Security isn't just about external attacks. Internal model degradation is a risk too.

Implement monitoring for:

• Sudden drops in inference quality (measured against a held-out evaluation set)
• Unusual token generation patterns (possible model poisoning indicators)
• Discrepancies between expected and actual model weights (hash verification on weight loading)

# Example: Weight integrity verification at startup
import hashlib
import json

def verify_model_integrity(model_path: str, expected_hash: str) -> bool:
    """Verify model weights against a known-good hash."""
    sha256 = hashlib.sha256()
    for root, dirs, files in os.walk(model_path):
        for file in sorted(files):
            filepath = os.path.join(root, file)
            with open(filepath, "rb") as f:
                while chunk := f.read(8192):
                    sha256.update(chunk)
    actual_hash = sha256.hexdigest()
    return actual_hash == expected_hash

Call this at container startup — before the model is loaded into GPU memory — and refuse to serve if the hash doesn't match. Log the event and trigger an alert.

Frequently Asked Questions

Q: Can I use GitOps with models that don't fit in a standard container registry?
A: Absolutely. Store model weights in a versioned object store (S3, GCS, etc.) with SHA-256 checksums, and have your deployment pull them at startup. The GitOps layer manages the pull logic and version references — not the weights themselves. Use init containers to verify checksums before the model server starts.

Q: How do I balance security hardening with inference latency?
A: Measure first. Network policies and seccomp profiles have negligible latency impact. TLS termination at the ingress adds ~1-2ms. The heaviest overhead comes from runtime security agents (Falco, Tetragon) — profile them in staging before production deployment. Consider offloading policy evaluation to the admission layer rather than runtime where possible.

Q: What's the minimal viable security posture for a small team?
A: Start with three things: (1) GitOps with a PR gate for all changes, (2) container-level network policies restricting egress, and (3) Kyverno/Gatekeeper policies enforcing non-root, non-privileged, and resource-limited pods. These three controls address the majority of production incidents. Add runtime security and model integrity verification once the foundation is stable.

---

Securing local LLMs in production isn't about adding more locks to a fragile door — it's about rebuilding the door itself. Self-hosted infrastructure gives you the raw materials. GitOps gives you the blueprint and the quality control. Together, they create a system where security is not a feature you bolt on, but a property that emerges from every layer, every commit, and every deployment.

For more on operationalizing AI systems at scale, check out Tamiz's Insights on the emerging patterns in production ML engineering.