Pipeline & Prompts | Byte size guides on DevOps, Cloud and AI
☁️ Cloud Without the Chaos #4
⚡ Byte Size Summary
- Why "the data is in an EU data centre" doesn't answer a sovereignty question. Support access, logs, and metrics are separate surfaces, and each needs its own boundary
- What happened when a blanket EU-only routing fix broke a 2-hour P1 SLA, and the tier we built instead
- Why sovereignty isn't something you automate. You can only shrink the surface area where a human has to touch the boundary
The Story
The data residency box was already checked. Managed Kubernetes clusters, multi-region, customer workload data pinned to EU regions from day one. That part of the compliance conversation had been closed for months.
Then a support ticket came in from an EU customer's compliance team, and it wasn't about where the data lived. It was about who could see it while fixing a problem.
Our support model at the time was 24x7 global follow-the-sun. A P1 ticket routed to whichever region had bench depth at that hour, which meant a customer's cluster could get worked by an engineer sitting in Asia at 3am their time. That engineer had platform-only access: control plane nodes and management-layer components, no customer workload data, no application logs, no application namespaces. Scoped correctly, by design, and had been for a long time.
The customer objected anyway. Not because anything had actually been accessed improperly — nothing had. The objection was on principle, GDPR-era: personal data protections don't stop being a concern just because the access is read-only and platform-scoped. Their compliance team wasn't comfortable with a data processing agreement that allowed any support engineer, anywhere, any kind of access path into infrastructure holding EU personal data, even platform-adjacent access with no visibility into the data itself.
That's the sentence that reframes the whole problem: "it's in an EU data centre" answers a data residency question. It says nothing about who can touch the systems around that data, from where, under what access model, logged how. Sovereignty isn't one control — it's one of five dimensions worth placing deliberately, and it's a set of surfaces in its own right: storage location, support access, logs, metrics, key management. Each one needs its own answer.
The Problem
Platform engineers and SREs running managed Kubernetes clusters for customers under multiple compliance regimes at once feel this pain first. GDPR, SOC 2, and industry-specific overlays on top — not SOC 2 alone. GDPR puts a human body in a specific relationship to EU personal data. SOC 2 wants an audit trail: who touched what, when. Industry-specific requirements, financial services and healthcare especially, stack their own access-logging and retention rules on top of both.
What breaks without an explicit answer to "who can access this, from where, under what conditions" is trust, not uptime. The systems keep running. The compliance team stops trusting the DPA. And once one EU customer's legal team flags a sovereignty gap, it doesn't stay contained to that account. It becomes a question every other EU prospect's procurement team asks in the next sales cycle.
The cost shows up as engineering time spent redesigning access models after the fact, a support organization now split along a line that didn't exist before, and a rollback path that isn't a policy flip. It's staffing and contract changes.
Why Existing Approaches Fall Short
The obvious first move is blanket EU-only ticket routing: if the customer is in the EU, route the ticket to EU-based support staff, full stop. We tried this. It broke the 2-hour P1 production SLA, because EU bench depth wasn't built for follow-the-sun coverage. It was built assuming tickets could route globally. A P1 that hits outside EU business hours with no EU engineer awake to take it either blows the SLA or forces someone to page a much smaller on-call rotation than the global model ever needed.
The other common move is to treat this as a policy document problem: write a stricter DPA, keep the access model the same. That doesn't survive contact with an auditor or a customer's own legal review. A written policy that says "support engineers only access what they need" isn't a control, it's a description of intent. GDPR and SOC 2 auditors want the boundary enforced in the access model itself: who has a credential, what that credential can reach, how long it's valid, what's logged when it's used. A policy without RBAC and logging behind it fails the first real audit.
Neither actually answers what the customer raised. Blanket geographic routing solves for where the person sits, not what they can access and for how long. A stricter policy document solves for looking compliant, not for being enforceable.
The Architecture
The diagram shows the EU sovereign support tier as a distinct zone from the global support pool, with the RBAC boundary (read-only default, short-lived-token break-glass) drawn explicitly between them, and the EU-boundary-contained data plane (logs, metrics, KMS keys) called out separately from the cluster control plane. What it needs to prove, not just show: two independent boundaries have to hold — an access boundary and a data boundary — not one.
The architecture that replaced blanket routing is sized per hosting environment, because not every managed cluster offering draws from the same support staffing pool. Different clusters can sit on different managed platforms with different regional engineer depth, so a single "EU tier" designed around one platform's bench doesn't automatically map onto another. Each hosting environment got its own EU sovereign support tier, sized to its own bench.
Staffing is business-hours-plus-pager-crew, not full 24x7 in-region rotation. Building a true 24x7 EU-only rotation for each platform would have meant hiring at a scale the customer base didn't justify yet. The pager crew covers the overnight P1 gap without requiring a full parallel global team inside the EU boundary.
Control plane. RBAC default is read-only for the EU-scoped support tier. Elevation beyond read-only goes through an automated break-glass path issuing short-lived tokens — no standing write access, no permanent elevated role sitting unused between incidents. Every break-glass elevation is itself an auditable event.
Data plane. Debug logs, metrics, and KMS keys used for this support tier stay entirely inside the EU boundary. This is the piece "the data is in an EU data centre" misses: it's not enough that the customer's application data stays in-region if the diagnostic exhaust generated while fixing their problem doesn't.
Knowledge gap. A smaller EU-scoped expert pool doesn't have the same depth of tribal knowledge as the full global team. An internal RAG system built from internal runbooks and past incident resolutions gives the EU pool a way to close knowledge gaps without routing questions outside the boundary.
Blast radius is the useful lens here. The failure mode the old model exposed wasn't a security incident — nothing was ever improperly accessed. It was an unbounded set of humans who could touch the boundary, with no way to prove the boundary held other than trusting intent. The new model bounds that set explicitly: fewer people, less standing access, shorter-lived elevation, every elevation logged.
Implementation
Prerequisites
- Separate RBAC groups per hosting environment for the EU sovereign support tier, distinct from the global support RBAC groups
- Break-glass tooling capable of issuing short-lived tokens tied to an incident ticket ID, not a standing credential
- EU-region log and metrics storage isolated from the global aggregation pipeline — an infrastructure decision, not just an access-policy one
- An internal knowledge base with enough historical runbook and incident content to make a RAG system useful to a smaller support pool
Step 1 — Scope the RBAC boundary per hosting environment
Default role for the EU support tier is read-only against control plane resources. No default write access, no default access beyond the control plane nodes and management-layer components already established under the original platform-only scoping.
Rollback consideration: removing this RBAC group is reversible on its own, but doing so without also reverting the log and metrics isolation in Step 3 leaves the boundary half-enforced. Access reverts to global scope while the data still lives in an EU-only pipeline nobody outside the EU tier can query during an incident.
Step 2 — Wire break-glass elevation to short-lived tokens
Elevation above read-only requires an incident ticket ID as input and issues a token scoped to a defined TTL, not a standing elevated role. The token expires; there's no cleanup step where someone has to remember to revoke it.
Rollback consideration: if break-glass tooling fails or is unavailable during an incident, the fallback can't quietly become "give someone standing write access." That recreates the exact problem this architecture exists to avoid. The fallback has to be a manual, logged, time-boxed exception process, not a bypass.
Step 3 — Isolate EU support logs, metrics, and KMS keys
Diagnostic data generated by EU-tier support activity — debug logs, metrics scraped during troubleshooting, keys used for that access — needs to land in EU-region storage, not the global aggregation pipeline the rest of support uses.
Rollback consideration: this is the expensive one to unwind. Once EU-only log storage exists and EU support workflows depend on querying it, reverting means either migrating that data's access model back to global or maintaining two parallel pipelines indefinitely. Not a flag flip.
Step 4 — Stand up the internal RAG system for the EU pool
Index internal runbooks, past incident write-ups, and platform-specific troubleshooting docs. Scope retrieval to what the EU support tier actually needs. This closes the tribal-knowledge gap between a smaller regional pool and the full global team without routing questions outside the boundary.
Rollback consideration: this piece is closer to a convenience than a compliance control. Removing it degrades EU-tier effectiveness but doesn't reopen the sovereignty gap on its own — it isn't carrying regulated data across the boundary.
Security Considerations
Break-glass is the attack surface, not the read-only default. A read-only RBAC default is easy to reason about. The real risk sits in the elevation path: if the break-glass tooling has a weak authentication step, or short-lived tokens get logged in plaintext somewhere they shouldn't, an attacker doesn't need to compromise the sovereignty boundary directly. They need to compromise the mechanism that grants exceptions to it. Every break-glass elevation needs its own audit trail, separate from the general access logs, because it's the highest-value target in this design.
Log and metrics isolation can leak through the RAG system if scoped wrong. The internal RAG system exists to close a knowledge gap for the EU support pool. But if it's indexed against a shared internal knowledge base that includes non-EU incident write-ups referencing customer-identifying details from other regions, the RAG system itself becomes an unintended cross-boundary data path. Scoping what gets indexed, and confirming incident write-ups used as retrieval material are scrubbed of cross-customer identifying detail, matters as much as scoping the RBAC.
Tradeoffs
What you gain: an enforceable, auditable answer to "who can access what, from where" that survives a GDPR or SOC 2 audit — an access model with logs behind it, not a policy document. You also gain a defensible sales conversation: the next EU prospect's compliance team gets a specific, technical answer instead of "trust our intent."
What you give up: SLA resilience during off-hours EU incidents. Business-hours-plus-pager-crew staffing is a real tradeoff against the old follow-the-sun model's bench depth. The EU tier will always have thinner overnight coverage than the global pool did, and that's a permanent operational cost, not a startup pain that goes away with scale.
Operability tradeoff: two support models now exist where one did before, and every team supporting these clusters has to maintain both. That's ongoing operational overhead, not a one-time migration cost. Every future support tooling change has to account for the EU tier's separate RBAC scoping and separate log pipeline, or it risks quietly breaking the boundary the whole architecture exists to hold.
What's Next
Cloud Without the Chaos #5 — Cost Profile and Demand Shape: When Cloud Elasticity Actually Pays Off
Dimension 3 of the original five-dimension framework, walked through with a real before/after cost model.
Written by Pipeline & Prompts | Byte size guides on DevOps, Cloud and AI
Top comments (0)