The EU AI Act is rolling out in stages. Prohibited practices since February 2025, GPAI rules since August 2025, high-risk system obligations from August 2026. Full enforcement across all risk categories hits in August 2027.
If you're running a proxy or gateway between your applications and LLM providers, some of this applies to you - maybe not in the way you'd expect.
Who is responsible for what?
The AI Act doesn't assign blame to a single layer. It distributes responsibilities:
- Model providers (OpenAI, Anthropic, or your self-hosted vLLM) handle model safety, training data, and GPAI obligations
- Infrastructure (your proxy/gateway) provides secure access control, usage tracking, and routing
- Your application owns the use case, risk classification, transparency, and monitoring
- End users use the service within the terms you set
A proxy doesn't train models or decide how outputs are used. But it's not a completely neutral pipe either - RBAC controls, rate limits, and routing decisions shape the compliance landscape. The legal classification of AI infrastructure providers is still being refined. This is where things stand as of Q2 2026.
The real problem: prompt logging
Here's where it gets practical. Most LLM gateways log prompts and responses. Some by default, some as an opt-in "observability" feature, some because it was easier to log everything than to think about what to exclude.
Under GDPR, prompt content becomes personal data the moment someone types a name, email address, or anything identifiable. Once your proxy stores that, you're on the hook for:
- A legal basis to process it
- Records of processing activities
- Data subject access requests (someone asks "what data do you have on me?")
- Right to deletion ("delete everything about me")
- Breach notification scope (a data breach now includes prompt content)
This isn't theoretical. If your proxy vendor stores prompts in a database, all of this applies to your DPA.
What the AI Act actually requires for logging
Article 12 requires "automatic recording of events" for traceability - but primarily for high-risk AI systems. The key word here is events, not content.
What the regulation expects: who made the request, when, which model, how many tokens, what happened. Metadata.
What most proxies do: store the full conversation. That's not what was asked for, and it creates obligations that weren't necessary.
| Capability | Relevant for compliance | Common in proxies |
|---|---|---|
| Usage tracking (who, when, which model) | Yes | Yes |
| Rate limiting per user/team | Recommended | Sometimes |
| Audit logs for admin actions | Yes (high-risk) | Rare |
| Model access control (RBAC) | Recommended | Basic or none |
| Prompt content logging | Not the baseline requirement | Often default |
| Cost tracking per team/org | Recommended | Sometimes |
In certain high-risk deployments, more detailed documentation may be required for incident investigation. But that obligation sits with the deployer's application layer - the proxy doesn't have the context to decide what needs to be recorded for a specific use case.
Data minimization as architecture
GDPR Article 5(1)(c) requires data minimization - collect only what you need. Article 25 requires Privacy by Design - build it into the system, don't bolt it on later.
For a proxy, this means: if you don't need prompt content, don't store it. Not "we have a toggle to disable logging" - there should be no logging code to toggle. The proxy reads the model field for routing, streams bytes to the provider and back, tracks metadata, and forgets the content.
This is the approach we took with VoidLLM. Not because the AI Act forced us - we made this decision before the regulation was finalized - but because zero-knowledge is the right architecture for infrastructure that handles other people's data.
The practical benefit: your Data Processing Agreement covers metadata only. No prompt content in scope means no content-related breach notifications, no content-related access requests, no content retention policies for the proxy layer.
What a proxy should give you
For AI Act readiness at the infrastructure level:
- Access control - org/team/user/key hierarchy so you know who has access to what
- Usage tracking - metadata per request, not content
- Rate limits and token budgets - constrain usage at every level
- Audit logs - track administrative actions (who changed permissions, created keys, modified models)
- Metrics - Prometheus or similar for monitoring and alerting
- Health checks - know when your upstream providers are degraded
What a proxy should not do
- Validate AI outputs (that's the model provider's or your responsibility)
- Detect prohibited use cases (that's governance, not infrastructure)
- Inject AI disclosure labels (your app handles user-facing transparency)
- Assess risk levels (the use case determines classification, not the proxy)
A useful principle
Log content where you understand and take responsibility for it. For most architectures, that's your application - where you have context about the use case, the user, and what needs to be recorded. Not your proxy, which sees bytes flowing through.
Some regulated industries may want content logging for their own reasons. That's a legitimate choice based on risk assessment. The point is it should be a conscious decision, not a proxy default you didn't know was on.
We're engineers, not lawyers. If you're evaluating proxies for a regulated environment, the architecture matters more than the feature list. We wrote a more detailed analysis with diagrams on our blog.
Top comments (0)