DEV Community: Nerav Doshi

IAM and Identity Across Clouds: Why Permissions Take Longer Than the Install

Nerav Doshi — Thu, 22 Oct 2026 19:57:16 +0000

Pipeline & Prompts | Byte size guides on DevOps, Cloud and AI

DevOps from Zero #12

⚡ Byte Size Summary

Tell the difference between authentication (who is this?) and authorization (what can it do?) on AWS, Azure and GCP

Map the same idea to each cloud's vocabulary: IAM roles, managed identities and role assignments, service accounts and IAM bindings

Add an identity pre-check to your runbook so approvals stop blocking installs

The Story

A ROSA install cannot start until the customer has accepted the terms and activated the ROSA listing on the AWS Marketplace. That part takes minutes. The part after it is where the time goes.

ROSA installs with an STS-based installer role. That role carries an elevated trust policy, so the customer's cloud security team has to review it before anyone creates it. On the engagements I worked, that review took 3 to 5 days. One customer took almost 2 weeks. Once a customer decided to proceed, the average was about 1 week from the start of the security review.

Nothing about the cluster was waiting. The team was waiting on a permission.

We fixed it with a runbook change, not a technical one. Identity and permissions approval became a required pre-check item. If the pre-check was not complete, the install did not start. The pre-check is a set of provider-specific scripts and CLI commands that inspect the target account or subscription for problems before anything gets provisioned.

I am telling you this in an identity article because it is the most common reason cloud projects stall, and it is rarely a technical reason.

The Problem

Every cloud has to answer two questions every time something makes a request:

Authentication: who is making this request?
Authorization: is that identity allowed to do this specific thing?

Beginners collapse these into one. They give a workload an identity, attach a role, and assume it can now reach everything it needs. Then the request fails with an access denied error, and they cannot tell which half is broken.

This happens on AWS, Azure and GCP. The names differ. The mistake does not.

The second problem is time. New cluster installs and new customer onboardings need elevated permissions: AWS service control policies (SCPs) that allow the install, and the equivalent on Azure and GCP through managed identities or service accounts. Those permissions need approval, and approval has a queue.

Why Existing Approaches Fall Short

The usual fix for a permissions error is to grant broader access until it works. An administrator-level role makes the error disappear. It also removes the review that the security team was about to do, and nobody can later explain why that identity needs that much.

The usual fix for the delay is to ask earlier, informally. That works until the person who remembers to ask is on holiday. A request that lives in someone's head is the same tribal-knowledge problem we covered in the Infrastructure as Code article, applied to permissions.

Neither approach is repeatable. Both lead to the same outcome: an install that starts, then stops.

The Architecture

The diagram shows the same three-step chain on each cloud: an identity, a grant that connects it to a role or permission set, and the resource it wants to reach. It marks the one place the chain breaks: the identity authenticates successfully, but no grant exists for the target resource.

That break point is the design lesson. A successful login proves nothing about permissions.

Concept	AWS	Azure	GCP
The identity	IAM role or user	Managed identity or service principal	Service account
The permissions	IAM policy	Role definition	IAM role
The grant	Policy attached to the role	Role assignment at a scope	IAM binding on a project or resource

How It Works: Step by Step

You can separate the two questions with commands. The first command on each cloud answers who am I. The second answers what can I do.

1. Check who you are authenticated as

# AWS
aws sts get-caller-identity

# Azure
az account show

# GCP
gcloud auth list

2. Check what that identity is allowed to do

# AWS: simulate whether a role may perform a specific action
aws iam simulate-principal-policy \
  --policy-source-arn <role-arn> \
  --action-names s3:GetObject

# Azure: list role assignments for an identity
az role assignment list --assignee <principal-id> --all

# GCP: show which members hold which roles on a project
gcloud projects get-iam-policy <project-id>

If step 1 succeeds and step 2 shows no matching grant, you have found the authentication-versus-authorization gap.

3. Put the check in the runbook

For ROSA, the pre-check also covers the account-level prerequisites before the install begins.

# [AUTHOR TO VALIDATE] — insert the pre-check script / CLI commands used in the standard runbook

The rule that mattered more than the script: the install does not start until the identity approval line is checked.

Security and Operational Considerations

Scope grants narrowly. On Azure, a role assignment at subscription scope reaches everything below it. Assign at the resource group or resource when you can.
Prefer attached identities over stored keys. Roles, managed identities and service accounts attached to a workload avoid long-lived credentials sitting in a repo or an environment variable.
Do not skip the review to save the week. The elevated trust policy on an installer role is the reason the review exists. Shorten the wait by asking earlier, not by granting more.
Rollback. Identity changes are easy to add and easy to forget. Record every role and policy you create for an install so you can remove them when the cluster goes away.

What Breaks at Scale

One account is manageable. Twenty accounts, subscriptions and projects are not, because the approval queue repeats per environment and manual grants drift apart. Three clouds also means three vocabularies for the same ideas, so a team fluent in IAM roles still has to relearn managed identities and service accounts.

The pre-check scales better than memory does. Each new environment runs the same list.

Quick Recap

Authentication and authorization are separate — a working identity does not mean a permitted action
The same chain exists on every cloud — identity, grant, resource; only the names change
Approval time is the real blocker — 3 to 5 days typical, almost 2 weeks at worst for the ROSA installer role
Make approval a pre-check — if it is not complete, the install does not start

Scored the Index Against Known Answers: 10 of 12, and What the Cap Really Costs

Nerav Doshi — Wed, 21 Oct 2026 15:34:47 +0000

Entry 17 ended on a problem I'd made for myself: I'd been judging rankings with nothing to check them against. This entry fixes that, at least a little. Twelve questions, each with one article that should answer it, plus two controls where the collection has no answer. I labeled each right article from the filenames and the descriptions in the repo index, not by rereading the articles, so the labels are my best read, not verified ground truth.

The script asks each question against all 741 chunks, finds the right article's best-scoring chunk, and ranks it two ways: among all chunks (what an uncapped query returns) and among distinct articles (what Entry 17's cap of one per document returns). It also counts how many of the top five chunks come from the right article. That last column matters more than it sounds. A cap can only move the right article up, because it removes chunks that sat above it, so document-level hit rates will always flatter the cap. What the cap costs is extra chunks from the right article, and only the chunk count shows that.

python3 eval_retrieval.py

#   doc_rank  chunk_rank  exp_in_top5  top1
1   1         1           5            retry-logic-and-tiered-alerting-github-actions.md
2   1         1           4            secrets-management-multi-cloud-pipelines.md
3   1         1           4            terraform-managed-openshift-state.md
4   1         1           5            database-migrations-zero-downtime-deploys.md
5   1         1           2            prompt-versioning-ci-openshift.md
6   1         1           3            swapping-llm-providers-without-rewriting-stack.md
7   5         8           0            ai-tooling-openshift-evaluation-framework.md
8   1         1           4            pitw-01-zero-downtime-deployments.md
9   1         1           5            ai-in-the-stack-02-rag-runbooks.md
10  1         1           3            ai-in-the-stack-06-n8n-workflows.md
11  1         1           3            what-are-containers.md
12  3         7           0            kubernetes-at-scale.md

Ten of twelve questions put the right article first, both ways. After four entries of staring at rankings I couldn't judge, that was the first real evidence the index mostly works.

The two misses both lost to broader articles. "How can an AI agent safely read from my Kubernetes cluster?" went to the AI tooling evaluation framework, with the MCP architecture article fifth among articles and eighth among chunks. "Should I run managed or self-hosted Kubernetes?" went to the Kubernetes-at-scale piece, with the managed-vs-self-hosted article third and seventh. So I printed the chunk that won each one. For the agent question, the winner (distance 246.5) comes from the framework's prerequisites section and says that any AI tool touching a cluster operates within or around OpenShift's security constraints and RBAC. That's on topic. It reads more like a second defensible answer than a retrieval error, and my label may simply have been too narrow. For the managed question, the winner (235.4) is a section of the Kubernetes explainer on the managed platforms you'll encounter, which says nobody runs Kubernetes completely from scratch in production. It matches the words "managed Kubernetes" but describes platforms rather than weighing managed against self-hosted, so that one looks like a real miss. In both cases I've read only the winning chunk, not the articles.

I'm keeping the original labels and the 10-of-12 figure. Relabeling the agent question now would raise the score based on what I saw, not on what I measured.

One worry going in was that the series' own entries about MCP, n8n and RAG, embedded in the same index, would outrank the real articles. At the top spot, none did, zero of twelve. The check only looks at first place, so they may still be crowding slots two through five. I haven't looked.

Now the cap, with the numbers to judge it:

	Uncapped (by chunk)	Capped at 1 per article
Right article at #1	10 of 12	10 of 12
Right article in top 3	10 of 12	11 of 12
Right article in top 5	10 of 12	12 of 12
Chunks from the right article in the top 5 (average)	3.17	1.00

The capped 1.00 follows from how the cap works: all twelve right articles land in the top five articles, and each can appear only once. Neither miss reached first place even with the cap, so it did nothing for the top result. What it did was pull the right article into a short list for the two questions that missed, which is where the top-3 and top-5 gains come from.

The cost shows in the last row. For the ten questions that worked, the uncapped top five held 3.8 chunks from the right article on average. Under the cap that drops to one. If what you want is a list of sources, the cap pulled the labeled article into view for both misses, though only one of them looks like a clear miss. If what you want is passages to hand a model, it would have discarded roughly three of every four chunks from the right article on the questions that were already working.

The controls behaved the way Entries 15 and 17 would predict. "How do I check pod status with oc?" returned Entry 08 at 225.6, and "What's the best pizza topping?" returned Entry 06, the entry that contains that exact question, at 448.4. Both nearest matches are write-ups that quote the query. And 225.6 looks like a very strong match for a question the collection doesn't answer, so a distance cutoff alone wouldn't have flagged it as "not in here."

Twelve questions is a small set. Each one moves a percentage by eight points, this was a single run, and the question wording and the labels are mine. Treat the 83% as a rough reading, not a benchmark.

This answers the question Entry 17 left open, narrowly: the cap is a trade, not a fix. It earns its place when the right article only needs to show up in a short list, and it costs a lot when the consumer needs depth. I haven't tested the middle, a cap of two or three per article, and with this eval set I now can. Before I do, I need to settle whether a question can have more than one acceptable article, and I want that decided by reading the articles, not by checking which labels would raise the score.

Tested a Per-Document Cap at Real Scale — and Found I Had No Ground Truth

Nerav Doshi — Thu, 15 Oct 2026 22:40:55 +0000

Entry 07 ended on a loose thread: a document split into 17 chunks crowds out equally relevant single-entry documents, just by having more shots at ranking. The usual fix is a per-document cap, so I flagged it and moved on. Now that the collection holds 741 real chunks instead of 20, it was time to find out whether the problem was real at this size, and whether the fix worked.

Measured first, before touching anything. Same question as Entries 06 and 15, ten results, no cap:

q = ollama.embeddings(model="nomic-embed-text", prompt="how do I check pod status with oc")
results = collection.query(query_embeddings=[q["embedding"]], n_results=10)

231.1  08-mcp-server-claude-code_chunk2
320.0  15-sse-reconnect-distance-instability_chunk1
325.4  mcp-server-architecture-platform-engineering-kubernetes_chunk12
328.6  13-hand-rolled-agent-loop_chunk2
334.6  mcp-server-architecture-platform-engineering-kubernetes_chunk19
344.5  13-hand-rolled-agent-loop_chunk4
347.5  13-hand-rolled-agent-loop_chunk3
350.6  mcp-server-architecture-platform-engineering-kubernetes_chunk16
353.3  mcp-server-architecture-platform-engineering-kubernetes_chunk13
357.4  08-mcp-server-claude-code_chunk1

Four documents fill all ten slots: four chunks from the MCP architecture article, three from Entry 13, two from Entry 08, one from Entry 15. That looks like the bias I expected, until you check which documents are missing. The three longest articles in the collection (the retry-logic piece and the Terraform state piece at 38 chunks each, the zero-downtime deploy piece at 37) don't appear at all. The winners are the topically closest documents, not the longest. Several chunks of one on-topic article can all be legitimately relevant, so some of this clustering isn't bias.

The top hit also looked too good. 231.1 is 89 points ahead of the runner-up, and when I printed the chunk, it was Entry 08's own quote of Claude Code's answer, including the phrase "checking pod status with oc". The index was retrieving the write-up of this exact question. That's a side effect of embedding the series' own entries into the store, and it fits the lexical-overlap suspicion from Entry 13: near-verbatim wording scores far better than anything merely related.

Then the cap. Fetch forty results, keep only the best chunk per source document, take the top ten:

raw = collection.query(query_embeddings=[q["embedding"]], n_results=40)
seen, kept = set(), []
for doc_id, dist, meta in zip(raw["ids"][0], raw["distances"][0], raw["metadatas"][0]):
    if meta["source"] not in seen:
        seen.add(meta["source"])
        kept.append((dist, doc_id))
    if len(kept) == 10:
        break

The top four were unchanged, since they were already four different documents. Slots five through ten changed completely:

	Uncapped, slots 5-10	Capped, slots 5-10
What fills them	More chunks of the MCP architecture article, Entry 13 and Entry 08	One chunk each from the chunking entry, secrets management, RAG runbooks, provider swapping, zero-downtime deploys, prompt versioning
Distance range	334.6 to 357.4	360.8 to 381.5

The cap removed extra chunks from articles that really are about Kubernetes diagnostics over MCP and replaced them with chunks from documents that look unrelated, judging by title and by distance. At this scale it traded relevant results for variety.

I had also been treating one result as the real test. Entry 02 was the top hit in Entries 06, 07, 13 and 15, and it vanished from the top ten here, capped or not, so I assumed the cap should bring it back. It didn't. Ranking its chunks across the whole collection shows where it went:

52   415.9  02-oc-cli-mentor-system-prompt_chunk1
73   423.6  02-oc-cli-mentor-system-prompt_chunk2
77   424.1  02-oc-cli-mentor-system-prompt_chunk0
137  441.4  02-oc-cli-mentor-system-prompt_chunk3
683  555.5  02-oc-cli-mentor-system-prompt_chunk4

Its best chunk scores 415.9, lower (better) than the 437.7 it scored when it ranked first among 45 entries. The document didn't get worse. Fifty other chunks now sit ahead of it, in a crowded middle where roughly forty chunks fall inside a band about 58 points wide. Entry 15 found that rewording one query moved a distance by 54 points, so that whole band could reshuffle under a different phrasing. I haven't tested that, but the numbers point that way.

The more important realization is that Entry 02 was never a correct answer. It's about system prompts, not pod-status commands, and Claude Code said so back in Entry 08. This query probably has no right answer anywhere in the collection, which means I was judging a ranking with nothing to check it against. Whether the cap "worked" isn't a question I can answer from this data.

So the finding isn't about the cap. The cap changes results, and at this size it costs relevant ones, but I can't call that good or bad without a question where I already know which document should win. The next entry builds that: a small set of questions, each with a known right document, checked by hand. Until then, treat every ranking in this series as a description of what the index returns, not evidence of what it should return.

Grew the Collection from 5 Entries to 741 Real Chunks

Nerav Doshi — Thu, 08 Oct 2026 15:10:28 +0000

Every retrieval test since Entry 05 has run against the same handful of entries — three documents in Entries 05-06, twenty after Entry 07's chunking. Realistic enough to prove the mechanics work, not realistic enough to trust the actual rankings. Growing this into something closer to the real published catalog was overdue, and getting there cleanly took two wrong turns first.

First attempt pointed the bulk-embed script at what looked like the right content folder and got back three files that were in Repo scaffolding, not articles — the real posts weren't there at all. Hugo commonly organizes content as page bundles, each article living in its own subfolder as <slug>/index.md, and the script's flat glob("*.md") only looked one level deep. Fixed by pointing at the actual posts directory directly rather than guessing at folder depth.

Second run picked up all 45 real files — the 15 "Today I Ran" entries plus 30 other published articles — and embedded 741 chunks. Looked like a clean win. It wasn't, quite: the first run's three junk files were still sitting in the collection, uncleaned, and worse, the 15 "Today I Ran" entries that were already in the store from Entries 05-06 got re-embedded under a different ID scheme than before. The original pipeline stored single-chunk files under their plain filename (02-oc-cli-mentor-system-prompt.md); this script's chunking always appends _chunk0, _chunk1, even for short files. Same content, two ID formats, both present at once — a self-inflicted version of the exact cross-document ranking bias Entry 07 found by accident with a real multi-chunk document.

Rather than hand-picking which of ~766 entries to delete, wiped the whole collection and rebuilt from nothing:

import chromadb
client = chromadb.PersistentClient(path="./chroma_db")
client.delete_collection("today_i_ran_notes")

Then re-ran the bulk script fresh. Clean this time:

Files found:      45
Files processed:  45
Files failed:     0
Chunks embedded:  741
Collection size:  0 -> 741

No duplicates, no scaffolding files, consistent chunking and ID scheme across every document. From 5 test entries to 741 real chunks drawn from the actual catalog, in one clean pass.

One thing to glag before next test: this collection now includes the "Today I Ran" series' own back catalog — Entries 01 through 15 — alongside everything else. The index can now retrieve passages from its own past entries about building the index. Querying this store about chunking bias will likely surface Entry 07, which is the entry that discovered chunking bias, embedded into the very store that bias affects. Recursive in a way that's more interesting than useful, but worth noting before the next retrieval test runs against this and something from the series' own history shows up as the top result.

The real payoff of this entry isn't the number 741 — it's that every retrieval test from here forward is running against something that resembles the actual size and shape of a real knowledge base, instead of a handful of hand-picked documents chosen to make a specific point work. Entry 07's chunking-bias fix (the per-document result cap flagged back then, still not built) matters a lot more now than it did against 20 entries.

Agentic AI Governance: Five Controls for Safe Write Access

Nerav Doshi — Tue, 06 Oct 2026 11:37:48 +0000

🤖 AI in the Stack #8

Pipeline & Prompts | Byte size guides on DevOps, Cloud and AI

⚡ Byte Size Summary

This article closes the governance gap Article 01 opened with — the gap between TP4 (observing systems) and TP5 (autonomous incident response) that no amount of model capability alone can close.

Five components are required before AI gets real operational authority: bounded blast radius, mandatory human checkpoints for state-changing actions, complete audit trails, rollback as a first-class operation, and a tested kill switch.

None of this is theoretical. Every component in this article has already appeared somewhere in this series — RBAC scoping from Article 03, human-in-the-loop approval from Article 07, the reason field in rollbacks from Article 04, and bounded retries from Article 06. This article is where they get assembled into one governance model.

The Story

Eight articles ago, I told you about sitting in a data scientist demo at IBM and feeling something click that wasn't comfortable. The models were good. The theory was sound. But almost none of it reached production, and underneath the impressive demos was data nobody had governed, bias nobody had checked for, and a gap between what we sold and what we could actually deliver safely.

That gap had a name, even if I didn't have the vocabulary for it at the time: governance. Not the bureaucratic kind. The engineering kind — the same discipline that stops a bad Terraform apply from taking down a cluster, the same discipline that makes a database migration reversible, the same discipline that means a bad deploy can be rolled back in minutes instead of hours.

This series has spent seven articles building toward AI systems with real operational capability. A RAG pipeline that answers from your actual runbooks. An MCP server that gives AI live access to your cluster. A prompt versioning system with audit trails and mandatory rollback reasons. A provider abstraction layer with automatic failover. An n8n workflow with bounded retries and explicit error handling, built after a runaway loop taught me exactly why those bounds matter. And the hard look at what happens when an agent gets a longer leash — read-only by design, a failed remediation test, stale training data masquerading as competence.

Every one of those articles was, in part, a governance article wearing a different costume. This is the article where the costume comes off.

The Problem

Article 01 introduced five touch points where AI changes outcomes in platform engineering — writing code, reviewing code, operating systems, observing systems, and responding to incidents. The diagram that accompanied it drew a hard governance gate between TP4 (observing) and TP5 (responding autonomously), with a deliberately blunt callout: the gap between TP4 and TP5 is not a feature gap, it is a governance gap.

The promise of agentic AI in infrastructure is TP5 — AI that does not just tell you a pod is failing, but restarts it; does not just surface an anomaly, but resolves it. That is where the real productivity gain lives, and it is also where a wrong action executes at machine speed before any human reviews it.

Closing that gap is not a matter of waiting for better models. It is a matter of building the same governance scaffolding around AI that platform engineering already builds around everything else with the power to change production state.

Why Read-Only AI Isn't Enough

Everything built so far in this series has lived comfortably in TP1 through TP4. RAG retrieval is read-only by construction. MCP tool calls in Article 03 were explicitly scoped to read-only RBAC permissions — "even if a bug in the tool definitions allowed a write operation to reach the Kubernetes client, the service account has no permission to execute it." The n8n workflow in Article 06 checks cluster state and notifies; it does not remediate anything itself.

That restraint was deliberate and correct as a starting point. It is also incomplete. The actual promise of agentic AI — the thing that justifies the investment, the thing that saves real engineering hours — is TP5. Autonomous incident response. Self-healing. The system that does not just alert you to a problem but resolves it.

Article 07 tested that leap directly. The remediation test failed — not because of RBAC or permissions, but because the reasoning behind each proposed action was built on stale knowledge. Two independent bad calls landed at the same time and looked like one cascading failure. The gap is not technical. It is governance.

The Architecture

The five components below form the minimum viable governance model for agentic AI with write access to OpenShift. Each component is a design constraint enforced in code, not a policy document or runtime configuration.

The five components assembled into a governance flow: Observation (TP4) feeds into a bounded agent → Proposal (RBAC-scoped) → Human Checkpoint → Kill Switch Check → Execute (full audit record) → Rollback plan pre-captured → Outcome logged.

Prerequisites

Before implementing these controls, you should have:

OpenShift 4.14+ (RBAC Role/ClusterRole APIs used here are stable across 4.13–4.16)
Cluster-admin or project-admin on the target namespace for RBAC setup
Familiarity with Article 03 (MCP server RBAC scoping) and Article 06 (bounded workflow execution)
A notification channel (Slack, PagerDuty, or equivalent) your team already monitors for approval requests

The Five Components of Safe Operational Authority

1. Bounded blast radius

Every capability an AI agent has should be scoped to the smallest possible domain of effect, by construction, not by instruction. This is the RBAC pattern from Article 03 generalised: a ClusterRole that grants get, list, watch is architecturally incapable of deleting anything, regardless of what the AI is told or how it is prompted. Telling a model "only restart pods in the staging namespace" is a request. Scoping its service account to only have permission in staging is a guarantee.

# The same pattern from [Article 03](/posts/mcp-server-architecture-platform-engineering-kubernetes/), extended to a bounded write capability
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: agentic-remediation-scoped
  namespace: staging  # Role, not ClusterRole — namespace-scoped by construction
rules:
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get", "list", "delete"]  # Delete triggers a managed restart via the controller
    # No access to secrets, configmaps, deployments, or any other resource type
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["get", "list"]

Blast radius is not a single property — it has at least three dimensions worth scoping independently: what the agent can act on (resource types), where it can act (namespace, cluster, environment), and how much it can do before stopping (rate limits, the same rate-limiting principle from Article 06, applied to write actions rather than just tool calls).

2. Mandatory human checkpoints for state-changing actions

This is the architecture decision Article 07 explored — read operations execute freely, but any non-read operation triggers a Human-In-The-Loop approval request that an administrator must explicitly clear before execution. The same instinct that Red Hat applied to OpenShift Lightspeed (read ops automatic, write ops gated) generalises cleanly beyond any single vendor's tool:

# Pattern: action proposal + approval gate, before execution
from enum import Enum
from dataclasses import dataclass
from datetime import datetime, timezone

class ActionStatus(str, Enum):
    proposed = "proposed"
    approved = "approved"
    rejected = "rejected"
    executed = "executed"

@dataclass
class ProposedAction:
    action_id: str
    tool_name: str
    parameters: dict
    proposed_by: str       # "agent" or a specific session identifier
    proposed_at: datetime
    risk_level: str        # low | medium | high — drives approval routing
    status: ActionStatus = ActionStatus.proposed
    approved_by: str = None
    rejection_reason: str = None

def propose_action(tool_name: str, parameters: dict, risk_level: str) -> ProposedAction:
    """
    The agent never calls the real tool directly.
    It calls this — which creates an approval record and notifies a human.
    """
    action = ProposedAction(
        action_id=generate_id(),
        tool_name=tool_name,
        parameters=parameters,
        proposed_by="agent",
        proposed_at=datetime.now(timezone.utc),
        risk_level=risk_level,
    )
    save_to_approval_queue(action)
    notify_approvers(action)  # Slack, PagerDuty, whatever your team already uses
    return action

def execute_approved_action(action_id: str, approver: str) -> dict:
    """
    Only called after a human has explicitly approved.
    This is the only path that reaches the real, state-changing tool.
    """
    action = get_action(action_id)
    if action.status != ActionStatus.approved:
        raise PermissionError(f"Action {action_id} is not approved")
    log_execution(action, approver)  # Audit trail — see component 3
    result = execute_real_tool(action.tool_name, action.parameters)
    action.status = ActionStatus.executed
    return result

The risk-level field matters operationally. Not every action deserves the same friction. A pod restart in a dev namespace might auto-approve after a short delay if nobody objects. A configuration change touching a production payment service should require explicit, named sign-off every time. Build the approval routing around risk, not around a single blanket policy — a system that requires the same heavyweight approval for everything trains people to rubber-stamp, which defeats the purpose.

3. Complete audit trails

Article 04's prompt registry made a reason field mandatory on every rollback — "the engineer to articulate the problem at the moment of the incident, not in a post-mortem three days later." Article 03 logged every tool call with timestamp, parameters, and calling identity before execution. The same discipline applies to every agentic action, end to end:

@dataclass
class AuditRecord:
    action_id: str
    tool_name: str
    parameters: dict
    proposed_at: datetime
    proposed_by: str
    approved_by: str
    approved_at: datetime
    executed_at: datetime
    result: dict
    outcome: str  # success | failure | rolled_back

The test for a sufficient audit trail is simple: if something goes wrong three weeks from now, can you reconstruct exactly what the AI proposed, who approved it, when it executed, and what happened as a result — without asking anyone to remember? If the answer requires someone's memory, the audit trail is incomplete.

4. Rollback as a first-class operation

Every action an agent can take needs a corresponding, tested rollback — designed at the same time as the action itself, not improvised under pressure when something breaks. This is the same principle behind terraform plan before apply, and behind the prompt registry's one-command rollback from Article 04.

@dataclass
class ReversibleAction:
    forward: callable      # The action itself
    reverse: callable      # How to undo it
    reverse_params_from: callable  # How to derive rollback params from forward's result

# Example: a pod restart is reversible by definition — Kubernetes recreates it
# Example: a scale-down is reversible — record current replica count before scaling
def scale_deployment(namespace: str, name: str, replicas: int) -> dict:
    current = get_deployment(namespace, name)
    previous_replicas = current.spec.replicas  # Capture for rollback BEFORE acting
    apply_scale(namespace, name, replicas)
    return {"previous_replicas": previous_replicas, "new_replicas": replicas}

def rollback_scale(namespace: str, name: str, previous_replicas: int):
    apply_scale(namespace, name, previous_replicas)

Some actions are not cleanly reversible — a deleted resource without a backup, a sent notification, a ticket created in an external system. Where true rollback is impossible, the governance answer is not to skip the rollback design step; it is to exclude that action from autonomous execution entirely. If you cannot undo it, a human approves it every time, with no exceptions carved out for convenience.

5. A tested kill switch

This is the component the runaway loop story in Article 06 made viscerally clear: a workflow needs a way to be stopped completely, immediately, without needing to understand why it is misbehaving first. For agentic systems, this needs to exist at the infrastructure level, not just the application level.

# A feature-flag-style kill switch, checked before every agentic action
apiVersion: v1
kind: ConfigMap
metadata:
  name: agentic-controls
  namespace: platform-tools
data:
  agentic_actions_enabled: "true"
  max_actions_per_hour: "20"
  emergency_stop: "false"   # Flip this to "true" to halt everything, instantly

def check_kill_switch() -> bool:
    """Called before every single agentic action. No exceptions."""
    config = get_configmap("agentic-controls", "platform-tools")
    if config["emergency_stop"] == "true":
        raise SystemHalted("Agentic actions are globally disabled")
    if not config["agentic_actions_enabled"]:
        raise SystemHalted("Agentic actions are currently disabled")
    return True

The kill switch needs to be tested before you need it — not discovered to be broken during the incident it was meant to stop. Run a drill. Flip it. Confirm the agent actually halts. An untested kill switch is a placebo, and you will not find out it does not work at a convenient time.

The Assembly

Putting all five components together, the architecture for safe TP5 operation looks like this:

Observation (TP4) detects anomaly
        │
        ▼
Agent proposes action — scoped by RBAC, bounded blast radius (1)
        │
        ▼
Risk-level routing → Human approval checkpoint (2)
        │
   ┌────┴────┐
   │Rejected │──→ Logged, agent learns nothing executes automatically
   └─────────┘
        │ Approved
        │
        ▼
Kill switch check (5) — still enabled?
        │
        ▼
Execute — full audit record written (3)
        │
        ▼
Rollback plan already captured (4) — ready if needed
        │
        ▼
Outcome logged, available for review

This is not a simpler version of full autonomy. It is the actual shape of safe operational AI — closer to a well-governed CI/CD pipeline with a manual production gate than to the autonomous-agent demos that dominate AI marketing. That is not a disappointing conclusion. It is the correct one, and it is achievable today, with tools this series has already built.

Security and Operational Considerations

Human approval bypass attack surface. The ProposedAction dataclass is the only path to execute_real_tool. If an attacker (or a compromised model) can call execute_approved_action directly, the entire governance layer is bypassed. Mitigate by running the agent and the approval system in separate trust domains — the agent's service account must not have permission to write audit records or approve its own actions. Enforce at the namespace and ClusterRole level, not at the application code level alone.

Audit trail completeness under failure. If log_execution() fails (e.g., the audit store is down), the action should be blocked — not executed with logging best-effort. Same as Article 03's rule: if you cannot log it, you cannot do it.

Kill switch propagation delay. A ConfigMap flip is processed by the controller on its next reconcile cycle — not instantly. In a crisis, that lag may be seconds too long. For critical environments, layer a Kubernetes NetworkPolicy that blocks egress from the agent namespace as a second, instant kill switch.

Approval fatigue under high alert volume. Once every write action requires human approval, a noisy anomaly detector can flood the queue. The risk-level routing in Component 2 must include automatic suppression for known transient conditions — otherwise, operators rubber-stamp under load, and governance becomes theatre.

What Still Cannot Be Automated Away

Two things in this architecture are not technical problems, and no amount of additional tooling resolves them.

Someone has to own the approval queue. A Human-In-The-Loop checkpoint is only as good as the human in that loop. If approval requests pile up and get rubber-stamped during a busy on-call shift, the governance model is theatre. This is an organisational commitment, not a configuration setting — the same way a code review process only works if reviewers actually review.

Someone has to decide what risk level means for your organisation. A pod restart might be low-risk for one team and high-risk for another, depending on what runs in that namespace and what the blast radius of a mistake actually is. This series can give you the scaffolding. It cannot make that judgement call for your specific environment — that requires the kind of contextual knowledge a platform team has and a generic framework never will.

What I'd Do Differently

If I rebuilt this governance model from scratch, I would start with the kill switch drill, not the approval queue. The #07 remediation test failed because two bad recommendations landed simultaneously and looked like one cascading failure — but we had no way to confirm "the agent is fully stopped" because we'd never tested the emergency_stop flag under load. An approval queue you can rubber-stamp through is a minor problem compared to a kill switch you can't trust.

I would also scope the initial agent to a single namespace from day one, not as a configuration toggle but as a ClusterRole that literally cannot address any resource outside it. The current implementation starts broad and narrows — safer to start narrow and widen, since the failure mode of "too narrow" is "agent asks for help," while the failure mode of "too broad" is "agent takes down prod at machine speed."

What's Next

Cloud Without the Chaos #6 — Blast Radius in Practice

Article 07 closed with a question: what does a safe path to write access actually look like? This article answered it with a five-component governance model. The next question is whether that model holds under real failure conditions — and that is exactly what Cloud Without the Chaos #6 will test, taking the five dimensions from CWTC #1 and running them through a live blast-radius exercise.

Browse the full AI in the Stack series to see how governance connects across the eight articles — from the evaluation framework through RAG, MCP, prompt versioning, provider abstraction, n8n orchestration, agentic safety, and now these five controls.

Written by Pipeline & Prompts | Byte size guides on DevOps, Cloud and AI

Cloud Fundamentals: VPCs and VNets Explained

Nerav Doshi — Mon, 05 Oct 2026 21:32:50 +0000

Cloud Fundamentals: VPCs and VNets Explained

Pipeline & Prompts | Byte size guides on DevOps, Cloud and AI

The Planning Call That Almost Went Sideways

I was advising a client on their cloud architecture as they planned an Azure Red Hat OpenShift (ARO) deployment. Early in the planning phase, someone pulled up the client's existing network address ranges next to the new environment we were designing.

They overlapped.

Nothing was broken yet — we caught it before anything got deployed. But it meant a Microsoft networking engineer had to get pulled into the conversation before we could move forward. I have seen the exact same pattern show up on AWS engagements too. Different cloud, same root cause: whoever set up the original network picked an address range without thinking about what it might need to connect to later.

That planning call is the reason this article exists. If you understand how VPCs and VNets actually work — and why overlapping address ranges are such a common trap — you will catch this before it costs you weeks of rework.

The Gated Neighborhood

A VPC (AWS, GCP) or VNet (Azure) is your own private, fenced-off section of the cloud provider's network. Nobody else can just wander in, and by default, nothing inside can be reached from the outside.

Think of it like a gated neighborhood:

The cloud provider is the country the neighborhood sits in
Your VPC or VNet is the gated neighborhood itself — your property line
Subnets are the streets inside the neighborhood
A gateway is the front entrance where traffic gets in and out
Security groups and firewall rules are the locks on each individual door

Diagram generated with AI assistance.

You do not get to build anything — a server, a database, a Kubernetes cluster — until you have decided where the streets go. That decision is your CIDR block (short for Classless Inter-Domain Routing) — the range of IP addresses your neighborhood is allowed to use.

This is exactly where my client's planning call ran into trouble. Two separate environments had each been handed the same kind of address range, without anyone checking whether those ranges would ever need to talk to each other.

Why Overlapping Address Ranges Break Everything

Here is the part that surprises people who are new to this: two networks with overlapping IP ranges cannot be connected. Not "connected with extra configuration." Cannot.

If your neighborhood and my neighborhood both have a street called "10.0.1.0," and we try to build a road between them, nobody's mail knows which house it is going to. Cloud routing works the same way. VPC peering, VNet peering, Transit Gateways — none of them can route traffic between two networks that claim the same address space, because the routing table cannot tell the two ranges apart.

This is not a rare edge case. It happens constantly during:

Company mergers and acquisitions, when two organizations' networks need to connect
Multi-cloud or hybrid-cloud projects, when an on-prem network needs to reach a cloud VPC
Managed OpenShift deployments, like the ARO example above, where the cluster's network has to coexist with whatever else the client already has running

Catching it during planning, like we did on that ARO engagement, means a design conversation. Catching it after deployment means re-architecting a live environment — the same kind of expensive, undocumented rework we covered in Infrastructure as Code, just one layer down the stack.

The Same Shape, Three Different Clouds

Every major cloud provider uses the same basic idea — an isolated network you carve into subnets — but the specific rules differ, especially once you add managed Kubernetes into the picture.

AWS — VPC
A VPC is regional. Each subnet inside it lives in a single availability zone, so you create separate subnets in different zones for redundancy, and decide which ones are public (reachable from the internet) and which are private.

Azure — VNet
A VNet works the same way conceptually. This is the environment from my client story — the CIDR block for the VNet has to be planned carefully before an ARO cluster ever gets deployed into it, because that address space has to coexist with whatever else the client is already running.

GCP — VPC (with a twist)
GCP's VPCs are global rather than regional, which is a real structural difference from AWS and Azure. Subnets are still regional, and you still reserve dedicated, non-overlapping ranges for whatever you deploy — the same planning discipline applies, just with a different top-level shape.

Three different providers, three slightly different rule sets — but the same underlying question every time: have you reserved enough non-overlapping address space for everything this network needs to talk to?

Quick Recap

Here is everything we covered today:

A VPC or VNet is your own isolated, fenced-off network inside a cloud provider — nothing gets in or out until you decide how
CIDR blocks define the address range your network is allowed to use, and they have to be planned before you build anything on top
Overlapping CIDR blocks cannot be connected — not with extra configuration, not with a workaround. Peering and routing tables cannot tell two identical ranges apart
This shows up constantly in mergers, hybrid-cloud connections, and managed OpenShift deployments like ARO
AWS VPCs and Azure VNets are regional and follow a similar subnet model
GCP VPCs are global rather than regional — a real structural difference, though the planning discipline is the same across all three

What's Next

Article 12 — IAM and Identity Across Clouds

Now that you understand how cloud networks are fenced off, the next question is who gets to walk through the gate. In Article 12 we cover IAM — the system that decides which users and services can actually do anything once they are inside your VPC or VNet.

Written by Pipeline & Prompts | Byte size guides on DevOps, Cloud and AI

Found this useful? Share it with someone about to plan their first cloud network — and save them the planning call I had to sit through. Follow along for a new article every week.

Reconnected Claude Code Over SSE — and Found the Distance Scores Aren't as Stable as I Thought

Nerav Doshi — Fri, 02 Oct 2026 20:54:31 +0000

Loose end from Entry 14: switching mcp_search_server.py from stdio to SSE broke the Entry 08 Claude Code connection, which was registered expecting a spawned process, not a network server. Fixing it turned out to be the easy part:

claude mcp remove today-i-ran-notes
claude mcp add --transport sse today-i-ran-notes http://localhost:8090/sse
claude mcp list

✔ Connected, server still running from Entry 14. Clean.

Re-ran Entry 08's original test — ask it to use the tool, search for the oc pod-status question. It worked, and gave a solid narrative answer identifying Entry 02's wrong command. But I wanted the actual distance score this time, not just the summary, so I asked for the raw tool output directly. That's where this got more interesting than "confirmed the reconnect works."

Two calls came back, for two slightly different phrasings of basically the same question:

Query	Distance to `02-oc-cli-mentor-system-prompt.md`
"how to check pod status with oc"	430.1
"oc get pods command output troubleshooting"	375.8

Same target document, same intent, a 54-point swing just from rewording. For comparison, the gap between the best and worst result within a single query was 442.6 → 461.1 — about 18 points. The variance from paraphrasing the question was three times larger than the variance between a genuinely-relevant result and a marginal one in the same result set.

That's a sharper version of the open question from Entry 13, and it resolves cleanly once you see both numbers side by side. Entry 13 guessed the embedding model might be rewarding lexical/syntactic overlap over pure meaning — a broken, command-shaped query scoring tighter than a clean one. This data points at something more basic underneath that: distance isn't calibrated for "is this relevant" at all, full stop. It's only meaningful for ranking results within one specific query's wording. Compare distances across different phrasings of the same question, and the number tells you more about word choice than about relevance.

There's also a reproducibility result worth stating plainly, since it cuts the other way: Entry 14's n8n test and this one both queried the same exact string — "how do I check pod status with oc" — through different clients, and both landed on 437.7. That's not in tension with today's finding. It's the other half of it: identical query text reliably gives identical distance, regardless of client or transport. Reword the query, even slightly, and the number moves more than you'd expect from meaning alone.

One more thing worth being honest about, since it affects how much to trust the narrative answer from the first run: Claude Code's accurate quote of the wrong oc command — the one from Entry 02 — didn't actually come from the MCP tool. The tool's results are truncated to ~200 characters per snippet, and that quote wasn't in any of them. The session separately grepped and read the full file directly off disk. So the earlier clean-looking answer wasn't a pure test of the RAG pipeline; it was the RAG pipeline plus Claude Code's own filesystem access filling the gap. Worth knowing before holding that result up as proof the retrieval system alone is doing the work — some of it was something else entirely.

Collection size matters here too, and it's worth naming rather than letting the clean distance numbers imply otherwise: every query in this entry returned the exact same five results, just reordered. The index doesn't have much in it yet. These rankings are real, but they're ranking a very short list.

Wired n8n to Ollama and the MCP Tool — and Hit Almost Every Wall on the Way

Nerav Doshi — Wed, 30 Sep 2026 01:09:19 +0000

n8n running as a container is a decent stand-in for "how would a real automation platform actually call this stuff". Everything up to now has been me calling things directly. Two integrations, same workflow: Ollama first, since it's simpler, then the MCP tool from Entry 08.

Ollama went cleanly. Container running, one HTTP Request node pointed at http://host.docker.internal:11434/api/generate — host.docker.internal matters here specifically, it's how a container reaches something running on the host machine itself, not localhost, which inside the container just points back at the container. Real response came back, done: true, no drama.

Except for the response itself, which is worth keeping verbatim: "To check the status of a Pod in Kubernetes using oc, you can use the kubectl command-line tool." Asked about oc. Answered with kubectl, immediately, in the same sentence that acknowledged the question was about oc. Same unconstrained-model bias from Entries 03 and 10, just a cleaner, funnier example of it this time — the model contradicts itself in one breath.

MCP was the actual project. I'd built the Entry 08 server over stdio — the transport where a client spawns your script as a subprocess and talks to it over stdin/stdout. n8n has a community node, n8n-nodes-mcp, that supports exactly that. Installed it, configured a credential, tried to execute:

Failed to execute operation: The file or directory does not exist

Worth stopping on this one rather than just patching around it, because the real cause is bigger than a wrong path. n8n runs inside its own container — a completely separate filesystem from the Mac. Pointing the STDIO credential at ~/mcp-env/bin/python3 was asking the container to find a file that, from its perspective, doesn't exist anywhere. And even mounting that path in wouldn't have actually fixed it: that venv's python3 is a macOS binary, and the n8n container runs Linux. A Linux container can't execute a macOS binary no matter where you mount it — that's not a path problem, that's a "these two things are architecturally incompatible" problem.

So: switched the server from stdio to SSE — a real network transport instead of a spawned subprocess — which sidesteps the whole cross-filesystem, cross-OS mess entirely. The container doesn't need to touch anything on the Mac's disk; it just makes a network request to a server the Mac is already running.

First attempt at that:

mcp.run(transport="sse", host="0.0.0.0", port=8090)

TypeError: FastMCP.run() got an unexpected keyword argument 'host'

Turns out host and port belong on the FastMCP() constructor in this SDK version, not on .run() — an easy mistake given how many things about this exact package have already moved around mid-series (the FastMCP → MCPServer rename from Entry 08 wasn't even the last surprise it had). Fixed, confirmed the server actually starts:

Uvicorn running on http://0.0.0.0:8090

Didn't assume the SSE path was /sse — checked directly instead:

curl -N http://localhost:8090/sse

event: endpoint
data: /messages/?session_id=f2000338ea514f89bcd2c695d56f49f7
: ping - 2026-09-28 20:28:31.952790+00:00

Real session, real pings. Confirmed.

Then n8n's side: Failed to execute operation: The service refused the connection. One more thing worth checking before assuming it was a config typo — this machine only has Podman installed, not Docker Desktop, even though the docker command works (Podman provides Docker-API compatibility). Whether host.docker.internal resolves the same way under Podman wasn't something I wanted to guess at, so tested it directly from inside the container:

podman exec -it n8n node -e "require('http').get('http://host.docker.internal:8090/sse', r => console.log('status:', r.statusCode))"

status: 200

It actually worked fine — Podman handles that hostname the same way Docker does here. So the earlier "connection refused" was something else, most likely a typo or leftover default in the node's endpoint field rather than a real networking gap. Fixed the field, re-ran.

Real result:

type: text
text: [02-oc-cli-mentor-system-prompt.md] (distance: 437.7) ...

437.7 — the exact same distance Claude Code got for this same query back in Entry 08. Different client, different transport, same underlying Chroma store, same number. That's about as clean a confirmation as this series has produced that the whole thing is actually wired together correctly, not just superficially working.

One correction on my own assumption before closing this out: I'd been telling myself we needed to switch to n8n's separate, official MCP Client Tool node once stdio was ruled out. Turned out unnecessary — the same community node that failed on STDIO also supports an SSE credential type directly. Same node, just a different saved credential, confirmed by opening the credential's own configuration panel directly rather than assuming from the node's label:

Looking back at the whole thing: none of the individual failures here were exotic. A wrong path, a renamed API, an untested assumption about hostname resolution, a mislabeled credential. What made this entry different from something like Entry 08 is that every layer — the model, the container runtime, the SDK, the automation tool — was a separate thing that could be individually right and still add up to a broken chain. Wiring four systems together doesn't multiply the failure points, it compounds them: each fix had to survive contact with the next layer before I actually knew it worked.

The one number that made all of it worth doing was that 437.7. Not because it's a good number — it's just a distance — but because it was the same 437.7 Claude Code got back in Entry 08, through a completely different path. That's the actual proof this series has been chasing since the RAG entries started: the same underlying system, reachable correctly from more than one direction, giving the same answer either way. Everything else in this entry was just the cost of getting there.

Kubernetes Cost Attribution: Namespace vs. Cost-Center

Nerav Doshi — Sun, 27 Sep 2026 13:42:38 +0000

Pipeline & Prompts | Byte size guides on DevOps, Cloud and AI

☁️ Cloud Without the Chaos #5

⚡ Byte Size Summary

Why native Kubernetes cost attribution (namespace/project-level, via CMMO) and organizational cost-center attribution are different problems, and why one mechanism can't do both jobs

How cost detection (fast, alert-based) and cost attribution (detailed, reconciled) are separate concerns that don't need — and shouldn't need — the same mechanism

What actually happens to attribution when the organizational-label layer goes away, and why that's not the same as losing all visibility

The Story

Cloud cost estimates tend to look straightforward when a workload is first being planned. Size the application, estimate average utilization, account for expected growth, enable autoscaling, and build the procurement number.

The problem starts when the workload doesn't behave like the estimate.

The procurement number for this engagement looked solid on paper. The application was homegrown, autoscaling was enabled, and the infrastructure had been sized against what appeared to be a reasonable average-utilization estimate. Nothing raised a concern during review, and there were no issues at go-live.

Then the cloud spend alerts started firing.

When we looked at the usage pattern by time of day, the reason became obvious. The heaviest workload wasn't happening during normal business hours. It was happening in the evenings, overnight, and during weekends — exactly the periods that had been smoothed into an average during the original sizing exercise.

Nothing unusual was happening with the application itself. API calls, logging, request processing, and the application's normal request/response activity were all behaving as expected. The autoscaler was doing what it was designed to do: responding to increased demand by adding capacity.

The problem was that the demand was occurring at a different time, and at a different shape, than the original estimate assumed.

The resulting cloud bill was approximately 20-30% higher than the original projection. And it stayed there. The workload's actual demand curve simply didn't match the curve used to build the estimate.

That gap between what we estimate and what the workload actually does is where the cost-attribution problem starts — the third of the five dimensions worth placing deliberately.

The Problem

The person who feels this first is usually whoever owns the cloud budget. The person who has to explain it is often whoever performed the original sizing.

That leads to a fairly simple question: which workload is actually driving the cost?

On a shared Kubernetes cluster running applications for multiple teams, knowing that the cluster cost a certain amount this month doesn't answer that question. Was the increase caused by one team's production application with genuinely spiky traffic? Was it a development or test workload that wasn't scaled down? Was a logging-heavy application consuming more resources than expected? Or was part of the cost associated with shared cluster infrastructure that shouldn't be assigned to an individual application team?

A cluster-level cost number doesn't provide that level of visibility.

Without granular attribution, the available responses tend to be broad ones: reduce the autoscaling ceiling, change the overall budget, or absorb the additional cost and try to improve the estimate next time. None of those approaches really solve the underlying problem.

Before we can optimize the cost, we need to understand where the cost came from.

Why Existing Approaches Fall Short

The first instinct is usually to look at the cloud provider's native cost-management capabilities. That's a reasonable place to start.

Cloud providers can associate costs with resources using tags, resource groups, subscriptions, accounts, and other infrastructure-level dimensions. This works well when the infrastructure resource and the accountability boundary are essentially the same thing.

Kubernetes changes that relationship. The cloud provider may be billing for virtual machines or node pools, while the application team is thinking in terms of namespaces, deployments, pods, and services. The infrastructure being billed and the workload consuming that infrastructure aren't necessarily the same object.

If three application teams share the same Kubernetes cluster, the cloud provider sees the infrastructure. The application teams see their workloads. Both views are correct. Neither view, by itself, answers the complete cost question.

The Architecture

Starting with native Kubernetes cost attribution

The next layer is understanding how much of the shared infrastructure each workload actually consumes. In an OpenShift environment, Red Hat Cost Management Metrics Operator (CMMO) provides this capability by using Prometheus/Thanos usage data from the cluster.

CMMO allows cost to be distributed at the project level, where an OpenShift project corresponds to a Kubernetes namespace. This is a significant improvement over looking only at the underlying infrastructure. Instead of asking how much did this cluster cost? we can start asking how much of that infrastructure consumption belongs to each project?

For many environments, that may be sufficient. In this case, it wasn't. The technical structure of the cluster didn't map cleanly to the organizational structure. A department could own several application namespaces. A namespace didn't necessarily map to exactly one department or cost center.

Namespace-level attribution correctly answers which workload or project consumed the resources? It doesn't necessarily answer which department or cost center should own that cost? That second question required another attribution dimension.

Adding organizational context

This is where labels become useful. Rather than replacing namespace-level attribution, we add organizational metadata to the workload. Deployment objects can carry labels representing information such as department, cost center, business unit, application owner, or environment.

An admission policy ensures the required attribution information is present before a deployment reaches production or is scheduled onto a node — see Implementation, Step 2, for the specific mechanisms this can be built on.

This creates two complementary dimensions. The namespace tells us where the workload lives. The label tells us who owns it from an organizational perspective. That distinction becomes particularly useful when the organization and the Kubernetes structure don't line up one-to-one. A department that owns five application namespaces can be shown per-namespace, or the label lets those five namespaces be viewed together as a single organizational cost center.

The label isn't replacing native Kubernetes cost attribution. It's adding context to it.

The two data sources

The implementation has two primary data sources.

The first is cluster usage data. CMMO reads Prometheus/Thanos usage data and uploads it to Red Hat's cost-management service on an approximately six-hour cycle. This provides the resource-consumption side of the equation: who used what?

The second is cloud billing data. For Azure, a native Azure Cost Export provides actual-cost data. In this implementation, the daily CSV export lands in a storage account in the same resource group as the cluster, and a service principal provides the access required for the cost-management platform to read that data. This provides the billing side: what did Azure actually charge?

The cost-management system correlates Azure VM instance IDs with OpenShift nodes and uses the usage information to distribute infrastructure costs across projects. The native attribution boundary is therefore the OpenShift project or namespace. The organizational label provides an additional dimension on top of that.

Two pipelines, not one

Cost attribution isn't a real-time process. There are two separate data pipelines that need to stay active:

Cluster usage: Prometheus/Thanos → CMMO → cost-management platform, on a roughly six-hour upload cycle
Azure billing: Azure Cost Export → storage account → service-principal-scoped access → cost-management platform, on a daily export cycle

As a result, the attributed cost view can take up to 24 hours to reflect the current state of the environment. The dashboard isn't a live query into the cluster — it's a view of the environment after the usage and billing data have passed through their respective collection and processing cycles.

Cost detection and cost attribution are different problems

The mechanism that tells us we have a cost problem doesn't need to be the same mechanism that tells us who caused it.

In this implementation, native Azure Cost Management spend alerts provide the faster detection mechanism — that's what identified the original overrun. The alerting path operates directly against Azure billing information and isn't dependent on the CMMO or Cost Export processing cycle.

Spend alerting asks has spending crossed the threshold? Cost attribution asks where did that spending come from? The first needs to be fast. The second needs to be detailed. Trying to make one mechanism do both jobs creates unnecessary complexity.

What happens when something goes wrong

Two failure points are worth naming.

The first is the cloud billing integration. If the service principal used to access the Azure Cost Export is given broader permissions than necessary, the integration could expose cost information beyond the intended scope.

The second is the workload-label enforcement mechanism. If the admission policy is bypassed or disabled, workloads may be deployed without the required organizational labels. That doesn't break namespace-level attribution — the native project/namespace cost information keeps working. What's lost is the additional department or cost-center dimension for workloads that don't carry the required metadata.

That's another reason to treat organizational labels as an additional attribution layer, not the foundation of the entire cost model — the foundation is CMMO's native namespace attribution, which keeps functioning whether or not the label layer does.

The diagram earns its place here by making one thing visible that prose can't: two independent pipelines, on two different cadences, converging on one dashboard — with the label/admission-policy layer drawn as an overlay on native namespace attribution rather than something the whole model depends on. That's the design decision worth seeing, not just reading.

The pattern itself isn't Azure- or ARO-specific: native platform-level usage attribution, paired with an organizational label layer for the cases namespace boundaries and org charts don't agree. Amazon EKS (Elastic Kubernetes Service) has its own usage/billing correlation through Cost and Usage Reports and Kubecost-style tooling; Azure AKS (Kubernetes Service) and Google GKE (Kubernetes Engine) have their own native cost management surfaces. The mechanics below — the operator, the export cadence, the specific IAM roles — are the ARO implementation of that pattern, because that's the engagement this comes from. Swap the platform-specific pieces and the same two-pipeline shape holds.

Implementation

Prerequisites

Azure Red Hat OpenShift (ARO) cluster with cluster-admin oc access
Azure subscription hosting the cluster, az CLI logged in with rights on the cluster's resource group
Red Hat Hybrid Cloud Console access with the Cloud Administrator role (or equivalent cost-management write access)
Kubernetes 1.30+ (or an OpenShift version that ships it) for native ValidatingAdmissionPolicy support — see Step 2 for the OPA Gatekeeper / Kyverno alternatives if you're not on a version that has it
Everything below (storage account, export, service principal) is scoped to the single resource group that holds the ARO cluster — not the whole subscription

Step 1 — Install the Cost Management Metrics Operator

# Verify the operator is available, then install via OperatorHub/Software Catalog
# or apply directly:
cat <<EOF | oc apply -f -
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: costmanagement-metrics-operator
  namespace: costmanagement-metrics-operator
spec:
  targetNamespaces:
    - costmanagement-metrics-operator
EOF

cat <<EOF | oc apply -f -
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: costmanagement-metrics-operator
  namespace: costmanagement-metrics-operator
spec:
  channel: stable
  installPlanApproval: Automatic
  name: costmanagement-metrics-operator
  source: redhat-operators
  sourceNamespace: openshift-marketplace
EOF

Then create the Hybrid Console service account (Settings → Identity & Access Management → Service Accounts, added to a group with the Cloud Administrator role), store its client_id/client_secret as a cluster Secret, and apply a CostManagementMetricsConfig:

oc create namespace costmanagement-metrics-operator --dry-run=client -o yaml | oc apply -f -

cat <<EOF | oc apply -f -
apiVersion: v1
kind: Secret
metadata:
  name: service-account-auth-secret
  namespace: costmanagement-metrics-operator
type: Opaque
stringData:
  client_id: "<CLIENT_ID>"
  client_secret: "<CLIENT_SECRET>"
EOF

cat <<EOF | oc apply -f -
apiVersion: costmanagement-metrics-cfg.openshift.io/v1beta1
kind: CostManagementMetricsConfig
metadata:
  name: costmanagementmetricscfg
  namespace: costmanagement-metrics-operator
spec:
  authentication:
    type: service-account
    secret_name: service-account-auth-secret
  packaging:
    max_reports_to_store: 30
    max_size_MB: 100
  prometheus_config:
    collect_previous_data: true
    context_timeout: 120
  source:
    check_cycle: 1440
    create_source: true
    name: aro-prod-cost
  upload:
    upload_cycle: 360
    upload_toggle: true
EOF

This engagement used the service-account auth type rather than the deprecated basic/token mode — worth calling out, since the token mode is what most quick-start examples default to. This is what produces native, project/namespace-level cost distribution from Prometheus/Thanos usage data — the baseline every other layer sits on top of.

Rollback here is a clean uninstall: delete the CostManagementMetricsConfig, then the Subscription and OperatorGroup, then the namespace. CMMO doesn't write anything back to the cluster's application workloads — removing it stops future uploads but doesn't touch anything already reported to the Hybrid Console.

Step 2 — Enforce the cost-center label at admission time

The requirement itself is simple to state: no Deployment reaches production or gets scheduled onto a node without a cost-center label present. This engagement enforced it with Kubernetes' own native admission policy — no external operator installed on the cluster.

What was actually used — native ValidatingAdmissionPolicy. CEL-based, GA from Kubernetes 1.30; confirm your OpenShift version ships it before relying on this path:

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: "require-cost-center-policy"
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
      - apiGroups: ["apps"]
        apiVersions: ["v1"]
        operations: ["CREATE", "UPDATE"]
        resources: ["deployments"]
  validations:
    - expression: "has(object.metadata.labels) && 'cost-center' in object.metadata.labels"
      message: "Deployments must include a 'cost-center' label."
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
  name: "require-cost-center-binding"
spec:
  policyName: "require-cost-center-policy"
  validationActions: ["Deny"]

No new controller to install or upgrade, at the cost of being the newest and least battle-tested option on OpenShift specifically.

If your cluster isn't on a version that ships native ValidatingAdmissionPolicy, or you're already standardized on a policy engine, the same requirement maps onto either of these — shown here for readers on a different cluster, not what this engagement ran:

Option — OPA Gatekeeper, via a constraint on K8sRequiredLabels:

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sRequiredLabels
metadata:
  name: require-cost-center-label
spec:
  match:
    kinds:
      - apiGroups: ["apps"]
        kinds: ["Deployment"]
  parameters:
    labels:
      - key: "cost-center"
        allowedRegex: "^[a-z0-9-]+$"

This assumes the underlying ConstraintTemplate for K8sRequiredLabels is already installed from Gatekeeper's constraint template library — the constraint above references it, it doesn't define it. Pin to Gatekeeper v3.x.

Option — Kyverno, via a ClusterPolicy:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-cost-center-label
spec:
  validationFailureAction: Enforce
  background: true
  rules:
    - name: check-cost-center
      match:
        any:
          - resources:
              kinds:
                - Deployment
      validate:
        message: "The 'cost-center' label is mandatory on all Deployments."
        pattern:
          metadata:
            labels:
              cost-center: "?*"

validationFailureAction: Enforce is what actually blocks the deploy — set to Audit first if you want visibility before you start rejecting anything.

No deployment artifact reaches a node without this label, regardless of which mechanism enforces it. Without it, Cost Management still gives you namespace-level attribution from CMMO — you just lose the department-level cut for any namespace that doesn't map cleanly to one cost center.

Step 3 — Export Azure billing data and grant read access

# Storage account in the same resource group as the cluster.
# public-network-access is disabled — reached only through a private
# endpoint, not a public IP. Where a private endpoint isn't an option,
# VNet rules plus IP firewall restrictions are the documented fallback.
az storage account create \
  --name "$CM_STORAGE" \
  --resource-group "$ARO_RG" \
  --location "$ARO_LOCATION" \
  --sku Standard_LRS \
  --kind StorageV2 \
  --min-tls-version TLS1_2 \
  --public-network-access Disabled

# Service principal for Red Hat's read access — needs BOTH roles
SP_JSON=$(az ad sp create-for-rbac \
  --name sp-rh-cost-management \
  --role "Storage Blob Data Reader" \
  --scopes "$EXPORT_SCOPE" \
  --json-auth)

az role assignment create \
  --assignee "$(echo "$SP_JSON" | jq -r .clientId)" \
  --role "Cost Management Reader" \
  --scope "$EXPORT_SCOPE"

Store the returned clientSecret immediately — it is not retrievable again after this command returns. Put it in Azure Key Vault if secrets are managed centrally, or in the CI/CD pipeline's own secure secret store (GitHub Actions Secrets, an Azure DevOps variable group) — never in a pipeline variable that lands in shell history or a plaintext log.

# CLI export creation needs the costmanagement extension and an existing container
az extension add --name costmanagement

az storage container create \
  --account-name "$CM_STORAGE" \
  --name costexport \
  --auth-mode login

STORAGE_ACCOUNT_ID="/subscriptions/${SUBSCRIPTION_ID}/resourceGroups/${ARO_RG}/providers/Microsoft.Storage/storageAccounts/${CM_STORAGE}"

# Recurrence start must be today or future — Azure CLI requirement
EXPORT_FROM=$(date -u -d '+1 day' +%Y-%m-%dT00:00:00Z)
EXPORT_TO=$(date -u -d '+2 years' +%Y-%m-%dT00:00:00Z)

# Daily actual-cost export, scoped to the resource group
az costmanagement export create \
  --name rh-cost-export-daily \
  --scope "$EXPORT_SCOPE" \
  --type ActualCost \
  --timeframe MonthToDate \
  --storage-account-id "$STORAGE_ACCOUNT_ID" \
  --storage-container costexport \
  --storage-directory daily \
  --recurrence Daily \
  --recurrence-period from="$EXPORT_FROM" to="$EXPORT_TO" \
  --schedule-status Active

If Red Hat Cost Management rejects the export schema, fall back to creating it through the Azure Portal instead — Cost Management + Billing → Cost export → Daily export, using the "Cost and usage details (actual)" template. That's the path the guide points to when the CLI-created export doesn't validate; worth knowing before spending an afternoon debugging a schema error the CLI doesn't explain.

The Cost Management Reader role matters as much as Storage Blob Data Reader — the read permission alone doesn't get you a functioning Azure integration in the Hybrid Console. Rollback is clean on this step in isolation: removing the export or the role assignment stops new billing data from landing, but doesn't touch what CMMO already reported from the cluster side.

Security Considerations

Service principal scope on the Azure billing integration. The service principal reading the daily cost export is, by definition, reading billing data that spans every namespace and every department on the cluster. If it's scoped more broadly than Storage Blob Data Reader + Cost Management Reader on the single resource group — subscription-level access, for instance — someone with access to the cost dashboard can infer more than cost: traffic volume and scaling patterns for departments other than their own. Confirmed as a real concern on this engagement; the mitigation is scoping strictly to the resource group holding the cluster, not the subscription.

Service principal credential storage. The az ad sp create-for-rbac command that provisions Red Hat's read access returns a client secret that Azure will not show again. Store it immediately in Azure Key Vault if secrets are managed centrally, or in the CI/CD pipeline's own secure secret store — GitHub Actions Secrets or an Azure DevOps variable group — so it never lands in shell history or a plaintext log.

CMMO authentication mode. This engagement used the Hybrid Console service-account auth type (client_id/client_secret stored as a cluster Secret) rather than the deprecated basic/token mode — worth stating explicitly, since token auth is what most quick-start examples default to and is the weaker of the two options for the cluster-to-console leg of this pipeline.

Read access on the cost dashboard itself is a separate control from the pipeline feeding it. Scoping the service principal correctly limits what the data pipeline can reach. It says nothing on its own about who can view the resulting cost report. This engagement handled that separately, through Red Hat Hybrid Cloud Console User Access Groups — only authorized finance and department leads had access to aggregated cost metrics, distinct from whoever manages the underlying Azure integration.

The storage account itself. The exported billing CSV sitting in the storage account is a sensitive, cross-department asset independent of anyone going through the Hybrid Console properly — someone with direct storage account access bypasses the Console's own access model entirely. This engagement secured it with Azure Private Endpoints, so the raw billing CSVs are never reachable over a public-facing IP; virtual network rules and IP firewall restrictions are the fallback where a private endpoint isn't an option.

Tradeoffs

What you gain / what you give up — granularity vs. currency. Two-layer attribution (native namespace/project plus label-based cost-center) gives you real per-application, per-department visibility that a flat cluster bill never could. What you give up is real-time accuracy: the ~24-hour cycle across CMMO's 6-hour uploads and Azure's daily cost export means the attributed-cost view is always behind. If you're making a same-day budget decision, you're making it against the faster but less granular native Azure Cost Management alert, not the project/label-attributed dashboard.

What you gain / what you give up — deployment friction vs. cost-center accuracy. Blocking unlabeled deployments at admission time is what makes department-level attribution work for namespaces that don't map cleanly to one cost center. What it costs is friction: every deployment pipeline now has a hard dependency on getting the label right, and a missing or wrong label doesn't just skew a report, it blocks a deploy. That's a deliberate tradeoff, not an accident, but it needs to be communicated to every team shipping to the cluster before it surfaces as a support ticket.

Operability tradeoff — two independent pipelines instead of one, with a graceful (not flat) failure mode. Running CMMO (cluster-side, usage-based) and the Azure Cost Export (billing-side, dollar-based) as two separate integrations that both have to stay active is more moving parts than a single source of truth would be. The upside of that separation: when the label layer goes down, attribution doesn't collapse to an even split across departments — it falls back to CMMO's native namespace/project view, which keeps working independently of the label/admission-policy layer. You lose the department-level cut, not all visibility.

What's Next

Cloud Without the Chaos #6 picks up the framework's fifth dimension directly: blast radius, tested honestly against a real failure instead of assumed on paper. This article's own two failure points — an over-scoped billing integration, and a bypassed label policy that degrades gracefully rather than catastrophically — are a small preview of that larger question.

There's no companion repository for this one. This documents a cost-attribution architecture and a set of operational decisions specific to one ARO engagement, not a general-purpose tool — the Gatekeeper/Kyverno alternatives in Step 2 are illustrative, not what was actually run here.

Written by Pipeline & Prompts | Byte size guides on DevOps, Cloud and AI

Built a Hand-Rolled Agent Loop and Found a Weird Retrieval Bias

Nerav Doshi — Thu, 24 Sep 2026 22:17:53 +0000

Every "agent" framework boils down to the same three-step loop underneath: look at the world, decide what to do, do it. I wanted to build that by hand instead of reaching for a framework, mostly because I've never actually seen the seams of one up close, and frameworks are good at hiding exactly the parts I wanted to look at.

Started with a completely fake skeleton, hardcoded task and all, just to prove the shape works before adding anything real:

def observe():
    return "task: find out how to check pod status with oc"

def decide(observation):
    return f"search_notes: {observation}"

def act(decision):
    print(f"Would call: {decision}")

obs = observe()
plan = decide(obs)
act(plan)

Fine, obviously — it's just string formatting at this point. The real test was replacing decide() with an actual model call:

import ollama

def decide(observation):
    prompt = f"""You are an agent with one tool available: search_notes(query).
It searches a local knowledge base and returns relevant notes.

Given this task: {observation}

Respond with ONLY the exact search query you'd pass to search_notes — nothing else, no explanation."""
    response = ollama.generate(model="llama3.2:1b", prompt=prompt)
    return response["response"].strip()

First two runs gave me the identical output, which made me assume it was deterministic. It wasn't — a third run proved that wrong immediately. Three runs, three different behaviors:

A full oc command wrapped in backticks — the model ignored the "respond with only a query" instruction entirely and answered the underlying question instead. Worth noting: it's the exact same flawed command from Entry 02 (--all-namespaces instead of scoping to the actual namespace).
find -f 'oc status pod*' -A -v — not a real command in any tool, just a strange mashup of find syntax and oc concepts.
How to check pod status with oc — the one time it actually did what was asked.

That instability is itself the finding. A "decide" step built on a small local model with no real constraint around it isn't reliable even for one fixed task run back to back — which is a pretty direct argument for why Entry 02's system-prompt approach exists in the first place.

Wired act() to the real MCP tool from Entry 08 next, to see what each of those three outputs actually retrieved when fed to real search — not through a full MCP client this time, just calling the underlying function directly:

import sys
sys.path.insert(0, ".")
from mcp_search_server import search_notes

def act(decision):
    result = search_notes(decision)
    print(result)

act('`oc get pods --all-namespaces -o jsonpath=\'{.items[*].metadata.name}\'`')
print("---")
act("find -f 'oc status pod*' -A -v")
print("---")
act("How to check pod status with oc")

Worth noting: search_notes is wrapped in @mcp.tool(), and I genuinely wasn't sure that would still work as a plain callable outside a real MCP session. It did — no errors, clean results, all three queries went straight through to the Chroma index.

And this is where it got genuinely strange. All three queries surfaced the same top document — 02-oc-cli-mentor-system-prompt.md, which makes sense, it's the only stored content actually about oc. But the distances ran backwards from what I expected:

Decide() output	Distance to correct match
Garbled backtick command	324.3 (best)
Nonsensical `find`/`oc` hybrid	378.7
Clean, correct query	430.1 (worst)

The most broken query retrieved the correct document with the tightest confidence. The one time the model actually did what I asked — produced a clean, sensible search query — retrieved the same correct document, but with noticeably weaker confidence than either of the broken attempts.

I don't have a confirmed reason for this, and I'd rather say that plainly than invent one. My best guess: Entry 02's actual content contains real oc command syntax, and the garbled query text — being made of similar command-like tokens — may be matching on surface-level lexical overlap rather than pure semantic meaning. That's a testable claim, not a settled one, and it's worth checking directly before trusting it.

So the loop works end to end — a real model decision, feeding a real search tool, returning real results. But this entry didn't end where I expected. The interesting result wasn't "the agent works," it's that the search layer built across Entries 05-08 might be rewarding queries that look like the stored content syntactically, more than queries that actually match its meaning. If that's true, it's a real weakness in the retrieval system itself, not just in this one flaky decide() step — worth its own entry to actually test properly.

What EU Data Sovereignty Actually Requires Beyond Hosting

Nerav Doshi — Mon, 21 Sep 2026 22:01:03 +0000

Pipeline & Prompts | Byte size guides on DevOps, Cloud and AI

☁️ Cloud Without the Chaos #4

⚡ Byte Size Summary

Why "the data is in an EU data centre" doesn't answer a sovereignty question. Support access, logs, and metrics are separate surfaces, and each needs its own boundary

What happened when a blanket EU-only routing fix broke a 2-hour P1 SLA, and the tier we built instead

Why sovereignty isn't something you automate. You can only shrink the surface area where a human has to touch the boundary

The Story

The data residency box was already checked. Managed Kubernetes clusters, multi-region, customer workload data pinned to EU regions from day one. That part of the compliance conversation had been closed for months.

Then a support ticket came in from an EU customer's compliance team, and it wasn't about where the data lived. It was about who could see it while fixing a problem.

Our support model at the time was 24x7 global follow-the-sun. A P1 ticket routed to whichever region had bench depth at that hour, which meant a customer's cluster could get worked by an engineer sitting in Asia at 3am their time. That engineer had platform-only access: control plane nodes and management-layer components, no customer workload data, no application logs, no application namespaces. Scoped correctly, by design, and had been for a long time.

The customer objected anyway. Not because anything had actually been accessed improperly — nothing had. The objection was on principle, GDPR-era: personal data protections don't stop being a concern just because the access is read-only and platform-scoped. Their compliance team wasn't comfortable with a data processing agreement that allowed any support engineer, anywhere, any kind of access path into infrastructure holding EU personal data, even platform-adjacent access with no visibility into the data itself.

That's the sentence that reframes the whole problem: "it's in an EU data centre" answers a data residency question. It says nothing about who can touch the systems around that data, from where, under what access model, logged how. Sovereignty isn't one control — it's one of five dimensions worth placing deliberately, and it's a set of surfaces in its own right: storage location, support access, logs, metrics, key management. Each one needs its own answer.

The Problem

Platform engineers and SREs running managed Kubernetes clusters for customers under multiple compliance regimes at once feel this pain first. GDPR, SOC 2, and industry-specific overlays on top — not SOC 2 alone. GDPR puts a human body in a specific relationship to EU personal data. SOC 2 wants an audit trail: who touched what, when. Industry-specific requirements, financial services and healthcare especially, stack their own access-logging and retention rules on top of both.

What breaks without an explicit answer to "who can access this, from where, under what conditions" is trust, not uptime. The systems keep running. The compliance team stops trusting the DPA. And once one EU customer's legal team flags a sovereignty gap, it doesn't stay contained to that account. It becomes a question every other EU prospect's procurement team asks in the next sales cycle.

The cost shows up as engineering time spent redesigning access models after the fact, a support organization now split along a line that didn't exist before, and a rollback path that isn't a policy flip. It's staffing and contract changes.

Why Existing Approaches Fall Short

The obvious first move is blanket EU-only ticket routing: if the customer is in the EU, route the ticket to EU-based support staff, full stop. We tried this. It broke the 2-hour P1 production SLA, because EU bench depth wasn't built for follow-the-sun coverage. It was built assuming tickets could route globally. A P1 that hits outside EU business hours with no EU engineer awake to take it either blows the SLA or forces someone to page a much smaller on-call rotation than the global model ever needed.

The other common move is to treat this as a policy document problem: write a stricter DPA, keep the access model the same. That doesn't survive contact with an auditor or a customer's own legal review. A written policy that says "support engineers only access what they need" isn't a control, it's a description of intent. GDPR and SOC 2 auditors want the boundary enforced in the access model itself: who has a credential, what that credential can reach, how long it's valid, what's logged when it's used. A policy without RBAC and logging behind it fails the first real audit.

Neither actually answers what the customer raised. Blanket geographic routing solves for where the person sits, not what they can access and for how long. A stricter policy document solves for looking compliant, not for being enforceable.

The Architecture

The diagram shows the EU sovereign support tier as a distinct zone from the global support pool, with the RBAC boundary (read-only default, short-lived-token break-glass) drawn explicitly between them, and the EU-boundary-contained data plane (logs, metrics, KMS keys) called out separately from the cluster control plane. What it needs to prove, not just show: two independent boundaries have to hold — an access boundary and a data boundary — not one.

The architecture that replaced blanket routing is sized per hosting environment, because not every managed cluster offering draws from the same support staffing pool. Different clusters can sit on different managed platforms with different regional engineer depth, so a single "EU tier" designed around one platform's bench doesn't automatically map onto another. Each hosting environment got its own EU sovereign support tier, sized to its own bench.

Staffing is business-hours-plus-pager-crew, not full 24x7 in-region rotation. Building a true 24x7 EU-only rotation for each platform would have meant hiring at a scale the customer base didn't justify yet. The pager crew covers the overnight P1 gap without requiring a full parallel global team inside the EU boundary.

Control plane. RBAC default is read-only for the EU-scoped support tier. Elevation beyond read-only goes through an automated break-glass path issuing short-lived tokens — no standing write access, no permanent elevated role sitting unused between incidents. Every break-glass elevation is itself an auditable event.

Data plane. Debug logs, metrics, and KMS keys used for this support tier stay entirely inside the EU boundary. This is the piece "the data is in an EU data centre" misses: it's not enough that the customer's application data stays in-region if the diagnostic exhaust generated while fixing their problem doesn't.

Knowledge gap. A smaller EU-scoped expert pool doesn't have the same depth of tribal knowledge as the full global team. An internal RAG system built from internal runbooks and past incident resolutions gives the EU pool a way to close knowledge gaps without routing questions outside the boundary.

Blast radius is the useful lens here. The failure mode the old model exposed wasn't a security incident — nothing was ever improperly accessed. It was an unbounded set of humans who could touch the boundary, with no way to prove the boundary held other than trusting intent. The new model bounds that set explicitly: fewer people, less standing access, shorter-lived elevation, every elevation logged.

Implementation

Prerequisites

Separate RBAC groups per hosting environment for the EU sovereign support tier, distinct from the global support RBAC groups
Break-glass tooling capable of issuing short-lived tokens tied to an incident ticket ID, not a standing credential
EU-region log and metrics storage isolated from the global aggregation pipeline — an infrastructure decision, not just an access-policy one
An internal knowledge base with enough historical runbook and incident content to make a RAG system useful to a smaller support pool

Step 1 — Scope the RBAC boundary per hosting environment

Default role for the EU support tier is read-only against control plane resources. No default write access, no default access beyond the control plane nodes and management-layer components already established under the original platform-only scoping.

Rollback consideration: removing this RBAC group is reversible on its own, but doing so without also reverting the log and metrics isolation in Step 3 leaves the boundary half-enforced. Access reverts to global scope while the data still lives in an EU-only pipeline nobody outside the EU tier can query during an incident.

Step 2 — Wire break-glass elevation to short-lived tokens

Elevation above read-only requires an incident ticket ID as input and issues a token scoped to a defined TTL, not a standing elevated role. The token expires; there's no cleanup step where someone has to remember to revoke it.

Rollback consideration: if break-glass tooling fails or is unavailable during an incident, the fallback can't quietly become "give someone standing write access." That recreates the exact problem this architecture exists to avoid. The fallback has to be a manual, logged, time-boxed exception process, not a bypass.

Step 3 — Isolate EU support logs, metrics, and KMS keys

Diagnostic data generated by EU-tier support activity — debug logs, metrics scraped during troubleshooting, keys used for that access — needs to land in EU-region storage, not the global aggregation pipeline the rest of support uses.

Rollback consideration: this is the expensive one to unwind. Once EU-only log storage exists and EU support workflows depend on querying it, reverting means either migrating that data's access model back to global or maintaining two parallel pipelines indefinitely. Not a flag flip.

Step 4 — Stand up the internal RAG system for the EU pool

Index internal runbooks, past incident write-ups, and platform-specific troubleshooting docs. Scope retrieval to what the EU support tier actually needs. This closes the tribal-knowledge gap between a smaller regional pool and the full global team without routing questions outside the boundary.

Rollback consideration: this piece is closer to a convenience than a compliance control. Removing it degrades EU-tier effectiveness but doesn't reopen the sovereignty gap on its own — it isn't carrying regulated data across the boundary.

Security Considerations

Break-glass is the attack surface, not the read-only default. A read-only RBAC default is easy to reason about. The real risk sits in the elevation path: if the break-glass tooling has a weak authentication step, or short-lived tokens get logged in plaintext somewhere they shouldn't, an attacker doesn't need to compromise the sovereignty boundary directly. They need to compromise the mechanism that grants exceptions to it. Every break-glass elevation needs its own audit trail, separate from the general access logs, because it's the highest-value target in this design.

Log and metrics isolation can leak through the RAG system if scoped wrong. The internal RAG system exists to close a knowledge gap for the EU support pool. But if it's indexed against a shared internal knowledge base that includes non-EU incident write-ups referencing customer-identifying details from other regions, the RAG system itself becomes an unintended cross-boundary data path. Scoping what gets indexed, and confirming incident write-ups used as retrieval material are scrubbed of cross-customer identifying detail, matters as much as scoping the RBAC.

Tradeoffs

What you gain: an enforceable, auditable answer to "who can access what, from where" that survives a GDPR or SOC 2 audit — an access model with logs behind it, not a policy document. You also gain a defensible sales conversation: the next EU prospect's compliance team gets a specific, technical answer instead of "trust our intent."

What you give up: SLA resilience during off-hours EU incidents. Business-hours-plus-pager-crew staffing is a real tradeoff against the old follow-the-sun model's bench depth. The EU tier will always have thinner overnight coverage than the global pool did, and that's a permanent operational cost, not a startup pain that goes away with scale.

Operability tradeoff: two support models now exist where one did before, and every team supporting these clusters has to maintain both. That's ongoing operational overhead, not a one-time migration cost. Every future support tooling change has to account for the EU tier's separate RBAC scoping and separate log pipeline, or it risks quietly breaking the boundary the whole architecture exists to hold.

What's Next

Cloud Without the Chaos #5 — Kubernetes Cost Attribution: Namespace vs. Cost-Center

Dimension 3 of the original five-dimension framework, walked through with a real before/after cost model.

Written by Pipeline & Prompts | Byte size guides on DevOps, Cloud and AI

Found the Real Kubernetes Memory Ceiling — It Wasn't Double

Nerav Doshi — Mon, 21 Sep 2026 22:01:01 +0000

Last entry left a real question hanging: the exact Podman number got killed under kind, twice. Did that mean the gap between the two runtimes was huge — needing something close to double the memory — or was the real ceiling only a little higher than 1663Mi, and I'd just clipped it? Only one way to find out.

Redeployed with the limit bumped to 2048Mi — a deliberately big jump, so the answer would be unambiguous either way:

kubectl delete pod ollama-sized
kubectl apply -f ollama-pod-2gb.yaml
kubectl exec -it ollama-2gb -- ollama pull llama3.2:1b
kubectl exec -it ollama-2gb -- ollama run llama3.2:1b

Asked it a real question, checked the outcome:

kubectl top pod ollama-2gb
kubectl describe pod ollama-2gb | grep -A5 "Last State\|Restart Count"

No crash. Restart Count: 0, and kubectl top showed real usage of 1775Mi — about 1.73GB. Which answers the question, and not the way I expected: that's only about 6.7% higher than Podman's 1.663GB. Not anywhere near double. The 2Gi limit worked, but it turns out to have been way more generous than actually necessary — the real ceiling sits much closer to 1663Mi than the size of my jump would suggest.

So that reframes last entry's open question. This wasn't a big structural gap between how Podman and Kubernetes handle memory. It was a narrow miss. Setting the limit at exactly a steady-state number, with zero margin for whatever memory spike happens during model loading before things settle down, was always going to be right on the edge of failing. Give it something like 10% headroom above the measured number, and it holds fine.

Which is really the actual lesson from this whole stretch of entries, more than "runtimes differ": a memory snapshot taken after a model is already loaded and idle isn't a safe number to use as a hard limit on its own. It doesn't account for the peak during loading. Going forward, my rule is measured steady-state number plus 10-15%, used as the real limit — not the raw number itself. 1663Mi measured should've meant something like 1875-1900Mi requested, and Entry 11's whole detour probably never happens.