Every prompt your application sends to a cloud AI provider is a data transfer. Under the GDPR, each one needs a legal basis, a documented purpose, and — when the servers sit outside the EEA — a Chapter V transfer mechanism. There is a cleaner architectural answer than stacking paperwork on every API call: run an open-weight model on infrastructure you control, inside the EU. What follows is a working setup plus the compliance checklist that makes it defensible — the things your counsel will ask about, not legal advice.
Why self-hosting changes the GDPR equation
When you call a third-party AI API, the provider is a processor acting on your instructions: you need a data processing agreement, you must check storage locations and sub-processors, and any transfer outside the EEA needs an adequacy decision or standard contractual clauses. The EU-US Data Privacy Framework's adequacy decision still stands — upheld by the EU General Court in September 2025 — but an appeal to the Court of Justice was still pending in October 2026, and two earlier frameworks were struck down. Planning for change is prudent, not paranoid.
Self-hosting an open-weight model removes the hardest questions by construction: no external processor sees the prompts, so there is no DPA, no sub-processor list, and no international transfer to manage for inference traffic; no vendor training risk, since paid API tiers already avoid training on API data but your own model makes it a non-question; and trivially true data residency — a server in Falkenstein or Roubaix keeps data in the EU without contractual gymnastics.
The honest trade: you take on running and securing the infrastructure, and you remain the data controller. The DPIA, the lawful basis, and the AI Act classification still apply. Self-hosting is not a compliance cheat code — it is a smaller blast radius.
The stack that works in 2026
As of October 2026 the current release is v0.40.3, the official ollama/ollama image supports NVIDIA CUDA, AMD ROCm, and a bundled Vulkan runtime, and every model the CLI serves is also available on a REST API — plus an OpenAI-compatible endpoint at /v1.
For models, the obvious starting point for EU teams is Mistral's open-weight models — an EU company, which removes the transfer question for the weights themselves. Llama 4, Qwen 3, and Gemma 3 are all in Ollama's library too. Licence-check every model you ship: "open-weight" is not "open-source", and commercial terms vary.
For hardware inside the EU, dedicated hosts from Hetzner (Nuremberg/Falkenstein), OVHcloud, or Scaleway are the standard choice. Rough sizing for 4-bit quantised models:
Model sizeRAM / VRAM neededRealistic hardware
8B (Q4_K_M)~6 GBAny modern VPS or laptop CPU
32B (Q4_K_M)~20 GBHigh-RAM dedicated server
70B (Q4_K_M)~40–48 GBGPU server (e.g. Hetzner's RTX PRO 6000 Blackwell box at €999/mo, per the Oct 2026 pricing)
Start with an 8B model on CPU — more than good enough for summarisation, classification, and structured extraction, the tasks where self-hosting wins. Scale up only when latency or quality forces you to.
A working setup in fifteen minutes
This Docker Compose file runs Ollama with model data on a persistent volume — the GPU block is commented out, uncomment it only on a GPU host:
services:
ollama:
image: ollama/ollama:0.40.3
container_name: ollama
restart: unless-stopped
ports:
- "127.0.0.1:11434:11434" # loopback only; expose via reverse proxy
volumes:
- ollama_data:/root/.ollama
# GPU variant: uncomment on a CUDA host
# deploy:
# resources:
# reservations:
# devices:
# - driver: nvidia
# count: all
# capabilities: [gpu]
environment:
- OLLAMA_HOST=0.0.0.0
volumes:
ollama_data:
Pull a model and verify the API responds:
docker compose up -d
docker exec ollama ollama pull mistral:7b-instruct
curl -s http://localhost:11434/api/tags | head -c 300
curl -s http://localhost:11434/v1/models | head -c 300
The second call is the OpenAI-compatible endpoint, so existing application code keeps working with one base-URL change:
from openai import OpenAI
client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")
resp = client.chat.completions.create(
model="mistral:7b-instruct",
messages=[{"role": "user", "content": "Extract the sender name from this email: ..."}],
max_tokens=64,
)
print(resp.choices[0].message.content)
Before this goes near real data: the port binds to loopback above; put any external access behind a reverse proxy with TLS and authentication. An unauthenticated LLM endpoint is a data leak waiting to happen — and under GDPR, it is your leak.
The GDPR checklist, item by item
1. Lawful basis, documented
For most internal tooling the basis is legitimate interests, with the balancing test documented before go-live. Consent is rarely right for employee-facing systems; contract performance only covers what the contract actually requires.
2. Data protection impact assessment
Run a DPIA before the system goes live, especially if outputs feed decisions about people or process sensitive data at scale. Do it first — it is where you record your answers to most of the other items.
3. Data minimisation in prompt construction
Strip what the model does not need before the prompt is built: no full customer records when a single field suffices, no raw email threads when a redacted extract works. Implement it in the code that assembles prompts — minimisation is a technical control, not a policy sentence.
4. Retention and erasure — including your own logs
Prompts, chat histories, and logs containing personal data are themselves personal data. Set a retention period and enforce it automatically — a nightly cron is worth more than a policy document:
# purge inference logs older than 30 days (adjust to your policy)
0 3 * * * psql "$DATABASE_URL" -c \
"DELETE FROM inference_logs WHERE created_at < NOW() - INTERVAL '30 days';"
Erasure requests must reach this store too. If a user asks for deletion, your logs, history, and caches must be covered — "the weights don't remember it" is not an answer.
5. Access control and encryption
TLS in transit, encryption at rest on the model-data volume, and rotated credentials. The Article 32 security obligations do not shrink just because there is no vendor to blame.
6. Audit logging — of access, not of content
Log who queried the system, when, and which model served them. Think twice before logging full prompt content: a content log is a second copy of the personal data, with its own retention and erasure obligations. Metadata or hashes instead is often enough.
7. Incident response
A breach of personal data on your LLM host triggers the 72-hour notification clock to your supervisory authority. Have a runbook: who revokes credentials, who assesses scope, who notifies. Nobody else's security team is watching the box.
8. Transparency
Your privacy notice must name the AI system and where data is processed. "Data never leaves our EU servers" is a strong, true statement for this architecture — make it, and keep it true.
9. Automated decision-making
If an output feeds a significant decision about a person by solely automated means, the GDPR's automated decision-making safeguards apply. In most cases the fix is a human-in-the-loop step — which is also the fix for model error.
10. EU AI Act deployer duties
The AI Act is now fully in force: deployers carry transparency duties — informing people when they interact with AI-generated content — plus incident reporting and technical documentation of the system.
What self-hosting does not solve
Training-data risk. The EDPB links unlawfully processed training data to deployment risk. A model trained on scraped personal data can memorise it; you inherit that exposure as the deployer. Ask how the model was trained and keep the answer in your DPIA.
Output risk. Hallucinated personal data about real people is still personal data you processed. Human review on consequential outputs is a compliance control, not just a quality one.
Licences. Weight licences vary; some restrict commercial use or redistribution. "Open-weight" is a technical fact, not a legal one.
Cost. Cloud APIs charge per token; self-hosting charges per month plus your time. At low volume, the API is usually cheaper. The honest case for self-hosting is compliance and data control, not the invoice — run the maths on your own traffic first.
Where to start this week
Stand up the CPU compose file on an EU host, run one real workflow through an 8B model, and write the DPIA from what you learn. The technical part takes an afternoon; the documentation takes a week. That ordering — working system first, then the checklist against the real system — is what keeps the paperwork honest. Self-hosting will not make you GDPR-compliant on its own, but it removes the questions you cannot answer and leaves only the ones you should be answering anyway.
Sources
Ollama v0.40.3 / Docker setup guidance — bitdoze.com, updated Oct 2026
Hetzner GEX63 Blackwell GPU server pricing and vLLM/Ollama deployment notes — qovery.com, Oct 17 2026
Self-hosted open-weight LLM architecture for GDPR (Art. 25, 32, 44) — dev.to enterprise LLM guide; promptquorum.com local-LLM privacy manifesto, Aug 2026
Controller obligations with self-hosted models: DPIA, lawful basis, automated decision-making assessment — januscompliance.co.uk, Oct 2026
"Is a self-hosted LLM automatically GDPR compliant?" — vdf.ai GDPR checklist, Oct 2026
Local LLM hardware discussion (Spark-X2.5 release, RAM vs VRAM debate) — social posts, Oct 9 2026
Originally published on LogicStack. Run the code examples right in your browser with our free interactive Python and SQL playgrounds — no setup, no sign-up.
Top comments (0)