DEV Community: Nerav Doshi

Managed Services vs Self-Hosted: When to Hand Over the Keys

Nerav Doshi — Fri, 21 Aug 2026 20:16:36 +0000

Managed Services vs Self-Hosted: When to Hand Over the Keys

☁ Cloud Without the Chaos — Article 02

You've placed the workload in the cloud. That decision is done. Now comes the one nobody gives you a clean framework for: do you run the infrastructure yourself, or do you hand the keys to a managed service?

I worked with a customer who answered this the hard way — twice. Once on their own platform, and once after they'd already made the switch. Both incidents involved the same underlying failure mode. Neither one turned out the way anyone in the room expected.

The warehouse analogy

In supply chain, this decision has a name: make vs. buy. Do you run your own warehouse, or use a third-party logistics provider (3PL)?

Running your own warehouse means you control everything — layout, staffing, hours, how fast you respond when a shipment needs to move at 2am. It also means you own every forklift breakdown and every understaffed shift.

A 3PL means someone else owns the building, the labor, the equipment. You focus on what you sell instead of how it's stored and moved. But you're also dependent on their systems, their SLAs, and their priorities when your shipment competes with someone else's for dock space.

Cloud infrastructure is the same trade in different clothes. Self-hosted is your own warehouse. Managed platforms are the 3PL. What follows is what that trade actually looked like for one team, in production, with real numbers attached.

The renewal that forced the question

The customer had a lean platform team self-managing Kubernetes — patching, logging, monitoring, upgrades, all of it in-house. Their platform renewal was coming up, expensive, and stacked against a team that was already stretched. The maintenance burden that made the renewal conversation painful had a specific shape: every major Kubernetes upgrade meant validating deprecated and removed APIs across their entire application estate, inside a fixed window, before the old control plane version went out of support.

That window was 30 days. The estate was roughly 1,400 microservices across 120+ namespaces. And on one upgrade cycle, something got missed.

The incident that made the case for managed

During a post-upgrade hotfix rollout, a Helm chart still referencing a removed Ingress API version hit the CD pipeline. The deployment failed. So the team did what you're trained to do — they rolled back.

The rollback failed too, and it failed for a specific, mechanical reason worth understanding rather than waving past. Helm doesn't recompute a prior release from source when you roll back — it retrieves the immutable historical manifest stored in a cluster Secret and re-applies it, which means that manifest still has to pass live API discovery validation on the way back in. Kubernetes 1.22 had removed the v1beta1 Ingress API group entirely. The API server had no way to accept a manifest written against an API version that no longer existed on the cluster. Helm halted before applying anything.

With the rollback path closed, the Ingress controller hit a state-sync failure. Payments traffic started returning 502s. Engineers resolved it by hand — removing orphaned resources, rewriting the manifests to the current API schema, and re-running the deployment.

Forty-five minutes of active, customer-facing outage on payments, before service was restored.

The team built a deprecation-scanning guardrail afterward. It was the right fix for the immediate problem. But it didn't touch the actual bottleneck, which wasn't technical — it was a small team with too many competing priorities trying to own an entire platform lifecycle by hand. That's what pushed the renewal conversation toward evaluating ROSA HCP and EKS as replacements, rather than just renewing what they had.

Why ROSA HCP over EKS

The deciding factor wasn't a feature checklist. It was a question of who owns which lifecycle. EKS would have meant the customer still assembling and operating most of the surrounding components themselves. ROSA HCP's integrated stack meant Red Hat owned Day 2 operations on the platform — the control plane, the surrounding components, the patching cadence — while the customer kept ownership of the application layer. For a team whose whole problem was "we can't keep operating all of this ourselves," that division of labor was the actual point — the same operational-ownership calculus that plays out anywhere a team decides what to stop managing directly.

The surprise: managed doesn't mean hands-off

Here's where the assumption cracked. The team expected that ROSA's managed Day 2 operations meant node draining, cordoning, and worker upgrades would happen the way a serverless abstraction behaves — invisibly, without touching running workloads.

That's not how it works. On ROSA HCP, control plane upgrades are decoupled and handled automatically by Red Hat. Worker node pool upgrades are not automatic in the same way — they require customer initiation, and once initiated, they're still governed by whatever workload configuration is sitting on top of those nodes.

A critical CVE forced the question. Red Hat's SREs patched the managed control plane within 24 hours of disclosure — the response time this engagement actually observed, not a contractual SLA figure. When the customer then initiated the worker pool upgrade, ROSA's automated drain process ran into a misconfigured PodDisruptionBudget (PDB) on a legacy service. The PDB was blocking pod eviction, so the node drain stalled.

The contrast that matters

This is the moment worth sitting with, because it's the clearest evidence for how this trade actually plays out.

On the self-hosted side, the team discovered their outage the way most self-hosted teams discover outages: customers saw 502s first. Forty-five minutes of active, visible damage before anyone had it under control.

On ROSA HCP, Red Hat's SRE automation caught the stalled drain and alerted the customer's team directly — before any workload was affected. The Machine Config Operator's drain process retried against the exponential backoff pattern it's designed to use rather than failing outright, so once the customer's application team fixed their own PDB configuration, the rollout resumed and completed on its own. No support ticket. No manual cluster-operator intervention from Red Hat.

That episode took 2 to 2.5 hours from alert to resolution — longer than the payments outage. But customer-facing impact was zero, because the misconfigured PDB had done its one useful job: it prevented running pods from being prematurely terminated while the drain was stuck.

Managed didn't mean faster. It meant the failure mode moved from "customer-facing outage you discover yourself" to "silent stall you get alerted to before anyone outside notices."

The exposure window nobody puts in the vendor deck

The CVE episode had a second layer worth naming plainly, because it's the part every "just move to managed" pitch tends to skip.

Red Hat's SREs patched the control plane in 24 hours. But because worker upgrades required customer initiation, and the customer held off starting that upgrade until their standard maintenance window — partly out of exactly the fear that then materialized as the PDB stall — the worker nodes running actual application workloads stayed unpatched for roughly four days. The 2.5-hour PDB delay was a small piece of that gap. The bigger piece was hesitation.

Managed infrastructure narrowed the attack surface. It didn't eliminate the scheduling risk sitting on the customer's side of the line.

What actually closes the gap

The fix wasn't anything Red Hat needed to change. It was proactive PodDisruptionBudget hygiene at the manifest level, applied consistently, so that node pool upgrades stop being a feared, manually-delayed event and become a routine background operation the team can trust. Once that trust exists, there's no reason to hold a CVE-driven worker upgrade for a scheduled maintenance window. The team's target shifted from a hesitant four-day window down to an automated 24-hour response following any control-plane security release — closing the exposure gap by removing the fear that created it.

Where this breaks at scale

At 120+ namespaces and 1,400+ microservices, PDB hygiene enforced only in CI/CD isn't enough. Pipeline linting has a blind spot: it can't see PodDisruptionBudgets generated at runtime by Operators, bundled inside third-party Helm charts, or applied directly via kubectl outside any pipeline at all. A policy that only lives in CI will pass clean while the cluster quietly accumulates exactly the kind of misconfiguration that stalled the worker upgrade in the first place. State drift on managed platforms follows the same pattern — the enforcement point that looked sufficient in isolation turns out to have blind spots once the estate gets large enough.

The customer's answer was to move enforcement to the cluster's front door — deploying Kyverno as a ValidatingWebhookConfiguration at the Kubernetes API server itself, so no PDB, regardless of where it came from, gets accepted onto the cluster without passing policy. It's deployed today. It hasn't yet been tested against a live attempted violation in production, so I'd call it the right architecture rather than a proven one — worth watching, not yet worth treating as settled.

The takeaway that doesn't show up in vendor documentation

The lesson here isn't "avoid managed services," and it isn't "always choose managed services" either. It's this: handing off infrastructure management doesn't reduce your governance burden. It relocates it, and intensifies it, at the API layer.

Once a vendor owns everything below the API — control plane, patching, node lifecycle — your job stops being "keep the cluster running." It becomes enforcing strict, API-level tenant policy so that the vendor's automation doesn't hit an invisible wall it has no way to see past. The vendor owns the warehouse. You still own what's on the shelves, and if what's on the shelves is stacked wrong, the automated forklift stops moving no matter how good the warehouse is.

A framework, not a rule

Three questions do most of the real work in this decision:

1. Is the scarce resource time, or is it control?
If the team is small and stretched, and the system isn't your competitive differentiator, lean managed. If it's core to what makes your product actually work, the operational knowledge from running it yourself has value beyond uptime.

2. Can you operate it today, under pressure — not eventually, today?
If not, that's not an argument for self-hosting. It's a sign you need training or headcount before this decision is even yours to make cleanly.

3. When it breaks, whose policy is actually holding it together?
Managed services don't remove your governance obligations. They just move the point of failure from "the platform" to "whatever policy your team enforces on top of it." Know which one you're actually testing before you sign the renewal.

None of these have a universal answer. They have an answer for your team, this quarter, with the staffing and policy discipline you actually have — not the version you're hoping to build eventually.

GitHub Repo

No companion repository for this article — this is a decision-framework piece about managed vs. self-hosted infrastructure, not an implementation walkthrough. Article 3 in this series returns to concrete implementation, where API-level governance becomes its own architecture decision.

What's Next

Cloud Without the Chaos, Article 03 picks up where this one leaves off: once you've handed the keys to a managed platform and enforced policy at the API layer, connectivity becomes the next architecture decision — VPNs, Direct Connect, ExpressRoute, and what actually breaks when two environments have to talk to each other reliably.

Built My First Local Embeddings Pipeline (Files + a Live URL)

Nerav Doshi — Fri, 21 Aug 2026 14:48:51 +0000

Context: An embedding model doesn't generate text — it converts text into a list of numbers (a vector) that represents the meaning of that text. Two pieces of writing about similar topics end up with similar vectors, even if they don't share any exact words, which is what makes semantic search possible: you can find "things that mean something like this," not just "things that contain this exact word." One detail that surprised me going in: the vector is always the same fixed length, no matter how long the original text is — a one-sentence note and a 10-page article both come out as the same-sized list of numbers, like a fingerprint that summarizes something much bigger into a fixed format. RAG (retrieval-augmented generation) is built on this: embed a bunch of documents, embed a question the same way, and find whichever documents are numerically closest to the question's meaning.

Ran: Built this up one command at a time instead of writing the full script upfront — easier to see what each piece actually does before combining them.

Step 1 — Pull the embedding model

ollama pull nomic-embed-text

Step 2 — Test the embedding API directly, before writing any Python

Ollama exposes embeddings over a local HTTP API, so you can sanity-check it works with a plain curl before involving any code:

curl http://localhost:11434/api/embeddings -d '{"model": "nomic-embed-text", "prompt": "hello world"}'

This returns a JSON object with a single embedding field — a list of 768 numbers. That's the whole idea of an embedding in one command: text in, fixed-length list of numbers out.

Step 3 — Install the Python pieces

pip3 install ollama chromadb requests beautifulsoup4 --break-system-packages

Step 4 — Embed one local file, interactively

Rather than run a full script blind, this is small enough to do a few lines at a time in a Python shell (python3):

import ollama
text = open("02-oc-cli-mentor-system-prompt.md").read()
response = ollama.embeddings(model="nomic-embed-text", prompt=text)
len(response["embedding"])   # → 768

Hit ModuleNotFoundError: No module named 'ollama' on the first attempt here — Step 3 hadn't been run yet. A reminder that even a five-step walkthrough has room to skip a step by accident.

Step 5 — Store it in Chroma

import chromadb
client = chromadb.PersistentClient(path="./chroma_db")
collection = client.get_or_create_collection(name="today_i_ran_notes")
collection.upsert(
    ids=["02-oc-cli-mentor-system-prompt.md"],
    embeddings=[response["embedding"]],
    documents=[text],
)
collection.count()   # → 1

Step 6 — Add a longer file, and hit a real limit

Repeating Step 4 against a longer draft article failed differently:

ollama._types.ResponseError: the input length exceeds the context length

nomic-embed-text has a 2048-token context window, and the longer draft (13,985 characters) blew past it. Quick fix for now — truncate to a safe length before embedding:

text = text[:6000]

Not the right long-term fix (more on that below), but enough to keep moving.

Step 7 — Add a live URL as a source

Pulling in a web page instead of a local file needs one extra step first: fetching the page and stripping out the HTML noise (nav bars, scripts, footers) so only the article text gets embedded.

import requests
from bs4 import BeautifulSoup

response = requests.get("https://pipelineandprompts.com/posts/03-1b-vs-3b-memory-comparison/",
                         headers={"User-Agent": "Mozilla/5.0"})
soup = BeautifulSoup(response.text, "html.parser")
for tag in soup(["script", "style", "nav", "footer", "header", "aside"]):
    tag.decompose()
article_text = soup.get_text(separator="\n", strip=True)
len(article_text)   # → 3359

That gets fed into the same ollama.embeddings() call from Step 4.

Result: Three sources ended up embedded across the two files plus the URL from the steps above:

Source	Type	Characters sent	Vector dimensions
`cloud-without-chaos-01.md`	local file	6,000 (truncated from 13,985)	768
`02-oc-cli-mentor-system-prompt.md`	local file	2,941	768
Entry 03 (live URL)	web article	3,359	768

Two things stood out. First, every vector came back at exactly 768 dimensions regardless of source length — confirms the "fixed-size fingerprint" idea from the context section wasn't just theory. Second, and more important: the truncation fix meant 57% of the longest draft's content (7,985 of 13,985 characters) never made it into its embedding at all. That's not a rounding error — over half the article is invisible to any future search against that vector. This is exactly why real RAG pipelines chunk long documents into smaller overlapping pieces instead of truncating: chunking keeps everything searchable, truncation just quietly throws away whatever didn't fit.

The URL fetch also worked cleanly — 3,359 characters extracted from the live page is close to the article's actual body length, suggesting the nav/footer/script stripping in the script did its job without pulling in much boilerplate.

Takeaway: The pipeline works end-to-end — files and live URLs, embedded into a queryable local store — but truncation is a real data-loss bug, not just a technical footnote, once you're working with anything longer than a short note. Next entry should replace it with actual chunking. Also still untested: whether querying this store actually retrieves the right document for a real question — that's the actual point of building this, and it's still ahead of us.

Compared Quantization Levels: Q4 vs Q8 vs FP16 on llama3.2:1b

Nerav Doshi — Wed, 19 Aug 2026 21:44:52 +0000

Context: A model's weights — the numbers it uses to reason — are normally stored at high precision, like FP16 (16-bit floating point). Quantization compresses those numbers down to fewer bits (Q8, Q4, etc.) to shrink the file size and memory footprint, trading some numerical precision for big space savings. It's a similar idea to compressing a high-resolution photo into a smaller JPEG: the file shrinks a lot, some detail is technically lost, but for most practical purposes it's hard to tell the difference.

Ran: Before pulling anything new, ran ollama show llama3.2:1b to check what Entry 01's baseline model actually was — and it turned out to already be Q8_0, not an unspecified "default" as I'd assumed back in Entry 01. Worth a correction: Entry 01 and Entry 03's 1b numbers were Q8, not full precision. Pulled two more variants for comparison: llama3.2:1b-instruct-q4_K_M (an explicit 4-bit quantization) and llama3.2:1b-instruct-fp16 (full, uncompressed precision). Loaded each in turn, sent the same test question, and captured ollama ps / ps aux for all three quantization levels.

Result:

	Q4_K_M	Q8_0 (Entry 01 baseline)	FP16
Download size	807 MB	1.3 GB	2.5 GB
`ollama ps` loaded size	997 MB	1.5 GB	2.7 GB
Process RSS	~0.94 GB	~1.24 GB	~2.57 GB

Memory scales cleanly with bit-width across all three: roughly 0.94 GB → 1.24 GB → 2.57 GB as precision doubles from 4-bit to 8-bit to 16-bit. That's a tidier relationship than I expected going in — quantization level is a solid, predictable lever for memory sizing.

For quality, I asked all three quantization levels the same real question: "what are good free AI tools that can create simple Google Slides using instructions?" None gave a clean answer. All three produced long lists mixing legitimate tools (Canva) with tools that have nothing to do with making slides — Q4 suggested Midjourney and DALL-E, FP16 added Artbreeder, Deep Dream Generator, and Prisma to the same mix. Full precision didn't fix this: FP16's list was just as padded with irrelevant suggestions as Q4's, and Q4 wasn't noticeably worse.

Takeaway: Memory scaled cleanly and predictably with quantization level — roughly doubling at each step from Q4 to Q8 to FP16, matching the bit-width math almost exactly. Answer quality told a different story: going all the way to full precision didn't reduce hallucinated suggestions on this open-ended recommendation question, and Q4 wasn't visibly worse than FP16. That's not proof quantization is "safe" in general — this was one soft question, not a rigorous eval — but it's a reminder that the real quality gap probably shows up on harder tasks (math, code, precise instruction-following), not casual recommendation queries. Also worth remembering going forward: check ollama show <model> before assuming what quantization a "default" pull actually gives you.

$ ollama show llama3.2:1b
  quantization        Q8_0

$ ollama show llama3.2:1b-instruct-q4_K_M
  quantization        Q4_K_M

$ ollama ps
NAME                         ID              SIZE      PROCESSOR    CONTEXT    UNTIL
llama3.2:1b-instruct-fp16    2887c3d03e74    2.7 GB    100% GPU     4096       4 minutes from now

$ ps aux | grep ollama
flyers  10763  0.3  15.7  438101456  2630128  ??  S  llama-server --model ... -c 4096
flyers   2256  0.0   0.3  436826160    56448  ??  S  ollama serve

$ ollama ps
NAME                           ID              SIZE      PROCESSOR    CONTEXT    UNTIL
llama3.2:1b-instruct-q4_K_M    22bc6b92eb01    997 MB    100% GPU     4096       4 minutes from now

$ ps aux | grep ollama
flyers  10786  0.2  5.7  436484688  963808  ??  S  llama-server --model ... -c 4096
flyers   2256  0.0  0.4  436826160   60400  ??  S  ollama serve

Agentic AI Infrastructure: What It Takes to Do It Safely

Nerav Doshi — Wed, 19 Aug 2026 03:00:06 +0000

Pipeline & Prompts | Byte size guides on DevOps, Cloud and AI

⚡ Byte Size Summary

See why we shipped an OpenShift diagnostic MCP server as read-only by design, and the RBAC wall that made write access harder than it looks

Walk through a real failed remediation test where an agent recommended a correct-looking fix built on stale, deprecated config — and what that failure mode actually is

Get the maturity-gated approval architecture we designed for write access — and why it's still sitting on paper, not in production

The Story

We built mcp-sre-tools — an MCP server that exposes OpenShift and Kubernetes diagnostics to an LLM, wired into Claude Desktop and n8n, covering ARO, ROSA HCP, OSD-GCP, and generic clusters. Nine diagnostic tools: get_cluster_health, diagnose_crashloop, get_failing_pods, and others in that family. READ_ONLY_MODE is on by default, and there are no write tools in the codebase at all. That part shipped clean.

The friction started when we scoped what came next: a remediation mode, where the agent wouldn't just diagnose a broken deployment — it would patch it.

That's where the story stopped being a build story and became an organizational one.

The Problem

Platform engineers and developers landed on opposite sides of the same question almost immediately, and for reasons that turned out to be more substantial than the usual risk-aversion reflex.

Developers were comfortable trusting agent-proposed changes roughly the way they'd trust a colleague's pull request — read the diff, sanity-check it, merge it. Platform engineers pushed back hard, and their objection wasn't reflexive. It was specific: a PR from a colleague comes with inspectable reasoning. You can ask them why. An LLM's proposed patch doesn't carry that same trail — the "why" is buried in a forward pass, not a code review comment.

Business stakeholders, meanwhile, were worried about something simpler and more immediate: an autonomous agent breaking a critical application in production.

Three legitimate concerns, three different vocabularies for the same underlying question — how much do we trust a system whose reasoning we can't fully inspect, applied to infrastructure we can't afford to break?

Why Existing Approaches Fall Short

The instinct is to reach for RBAC and call it solved. Scope the agent's service account to a namespace, give it patch permissions on Deployments and nothing else, and let it operate inside a fence.

That fence has a hole in it. Meaningful remediation almost always eventually needs to touch Secrets or environment variables — a misconfigured database connection string, an expired credential reference, a missing env var causing a crash loop. The moment your remediation scope includes Secrets, "namespace-scoped RBAC" stops being clean sandboxing and starts being a much bigger trust surface than the phrase implies.

We didn't have a way around that with RBAC alone. So we fell back to a narrower, honest justification for read-only: even without write access, a diagnostic agent cuts human mean-time-to-resolution. It's a smaller value proposition than full self-healing, but it's a real one — and it's the one we could actually defend without hand-waving.

The Architecture

Diagram 1 — as built: the shipped, read-only MCP architecture.

The controls we designed the server to work with — note the repo intentionally ships without a default rbac.yaml, to stay adaptable across cluster types and org policies. Deployment teams are expected to write their own scoped ClusterRole/RoleBinding tailored to their access model; the sample below shows the shape we recommend, not a default that ships:

Namespace-scoped RBAC (recommended, not shipped) — bind the MCP server's service account to Role/RoleBinding resources scoped per-namespace, not a cluster-wide ClusterRole
Service-account-based access — no static kubeconfig or personal credentials in the agent's execution path
NetworkPolicy egress/ingress restriction — the MCP server's pod network is fenced to only the cluster API and the LLM endpoint it needs to reach
Logging and observability as non-functional requirements — every tool call is logged, not bolted on after the fact

# Recommended shape, not a shipped default — deployment teams write their own
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: mcp-sre-tools-reader
  namespace: <target-namespace>
rules:
  - apiGroups: [""]
    resources: ["pods", "events"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["apps"]
    resources: ["deployments", "replicasets"]
    verbs: ["get", "list", "watch"]

Diagram 2 — proposed, never implemented: the maturity-gated write-access flow.

⚠️ This is a design artifact, not a shipped system. The kill switch and rollback automation shown below were never built.

  ┌─────────────────────┐
  │   Anomaly detected   │
  └──────────┬───────────┘
             │
             ▼
  ┌─────────────────────────────┐
  │  Agent proposes RBAC-scoped  │
  │      remediation action      │
  └──────────┬───────────────────┘
             │
             ▼
  ┌─────────────────────────────┐
  │      Risk-based routing      │
  │  low risk        high risk   │
  └──────┬───────────────┬───────┘
         │               │
         ▼               ▼
  ┌─────────────┐ ┌─────────────────────┐
  │ Light review │ │  Human approval gate │
  └──────┬───────┘ └──────────┬───────────┘
         │                    │
         └─────────┬──────────┘
                    ▼
        ┌───────────────────────┐
        │   Kill-switch check    │  ◄── [NEVER BUILT]
        └───────────┬─────────────┘
                    │  pass
                    ▼
        ┌───────────────────────┐
        │  Execute + full audit  │
        │        record          │
        └───────────┬─────────────┘
                    │
                    ▼
        ┌───────────────────────┐
        │ Pre-captured rollback  │  ◄── [NEVER BUILT]
        │   plan (ready before   │
        │      execution)        │
        └───────────┬─────────────┘
                    │
                    ▼
        ┌───────────────────────┐
        │    Outcome logged      │
        └───────────────────────┘

The flow we designed, on paper, for a future write-capable mode:

Anomaly detected
Agent proposes an RBAC-scoped remediation action
Risk-based routing — low-risk actions get lighter review, high-risk actions route to human approval
Kill-switch check before execution
Execution, with a full audit record
Pre-captured rollback plan available before the action runs, not written after
Outcome logged

Two components in that list were never built: the kill switch and the rollback automation. They exist as a design, not as shipped capability. Worth saying plainly, because it's the honest state of the project, not a gap we're hiding.

How It Works: Step by Step

For the part that is live — the read-only diagnostic path:

An alert or a manual query triggers the agent via Claude Desktop or an n8n workflow
The agent calls one or more of the nine MCP tools (e.g., diagnose_crashloop) against the target cluster
The MCP server's service account, scoped by namespace RBAC, executes the read-only API calls
Results return to the LLM, which synthesizes a diagnosis
A human reads the diagnosis and decides what to do next — the agent stops there

For the part that stayed on paper — the proposed remediation path — see the seven-step flow above. It never advanced past step 2 in production; steps 3 through 7 are design artifacts.

Security and Operational Considerations

RBAC and least privilege. The repo ships without a default rbac.yaml by design, to stay adaptable across cluster types — deployment teams write their own scoped ClusterRole/RoleBinding for their environment. That's also where the write-mode proposal broke down: remediation that needs to touch Secrets can't stay inside a tidy read-only-style RBAC boundary no matter who authors it.

Secrets exposure. The live tool set never reads or writes Secret contents. The unbuilt remediation mode is exactly where that boundary would have been tested, and wasn't.

Blast radius. Contained by design in the shipped version — nine read-only tools can't mutate cluster state, full stop.

Rollback strategy. For the live tools: not applicable, nothing is mutated. For the proposed write mode: rollback was designed as a pre-captured plan, generated before execution rather than reconstructed after a failure — but this was never implemented, and the one real recovery event we had (below) had to be handled manually.

Auditability. Every diagnostic tool call is logged. The proposed write-mode flow adds a full audit record as one of its seven stages — again, unbuilt.

What Breaks at Scale

We ran a private test — outside the public repo — with write/patch/upgrade-capable variants of the agent, specifically to see whether the self-healing story held up. It didn't, and the way it failed is the most important finding in this whole project.

The agent correctly diagnosed a real problem. Then it recommended a fix built on fluentd-era OpenShift Logging configuration — the collector layer — against a cluster already running the current Vector-and-Loki stack. The recommendation looked plausible. It targeted a collector and CRD shape that the cluster had already moved past.

At close to the same time, in the same test, the agent made a second, unrelated bad call: a recommendation to change the cluster's node-level hardware settings via the worker MachineSet. That recommendation, applied, provisioned the wrong instance type.

Two independent, unrelated bad recommendations landing at the same time — one on the logging stack, one on node hardware — made it look, in the moment, like a single cascading failure. It wasn't. Untangling that during recovery took longer than fixing either problem alone would have, precisely because the two failures got conflated. Recovery meant two separate fixes, not one clean GitOps revert: an updated ClusterLogging CR to throttle ingress at the collector layer and redirect output pipelines directly to the Loki backend, and a reconciliation of the worker MachineSet back to its approved providerSpec baseline — scaling down the misconfigured nodes and letting the Machine Operator provision replacements.

The label that matters here isn't "hallucination" in the dramatic sense people usually mean. The logging recommendation wasn't nonsense — it was training-data staleness masquerading as competence: correct-sounding reasoning built on a collector and config shape the cluster had already moved past. That's a more insidious failure mode than the black-box framing usually implies, because the output looks exactly as confident whether the underlying knowledge is current or stale.

The instance-type recommendation was a different kind of mistake — not stale knowledge, just a bad call. What made the incident harder to diagnose wasn't either failure alone, but the two landing together and looking, briefly, like one problem instead of two.

Scaling this up doesn't just mean more permissions — though it does mean that, and the platform team's original objection was concrete and correct on that point alone. It also means the bottleneck moves. Once you gate every write action behind human approval, the approval step itself becomes the constraint, and approval fatigue doesn't scale linearly with the number of things an agent proposes.

Quick Recap

Read-only shipped, write-mode didn't — and the RBAC wall around Secrets access is a real reason, not a cautious excuse
Two unrelated bad calls can look like one big failure — a stale-knowledge fix and a separate bad hardware recommendation landed together and were harder to untangle than either would've been alone
Governance architecture is easy to design and hard to ship — the kill switch and rollback automation are still just a diagram

GitHub Repo

Full implementation: mcp-sre-tools

What's Next?

The open question this leaves for the series: what does a safe path to write access actually look like, given that the blocker wasn't RBAC mechanics but the reliability of the reasoning behind each proposed action? That's worth its own piece.

Written by Pipeline & Prompts | Byte size guides on DevOps, Cloud and AI

Compared llama3.2:1b vs llama3.2:3b Memory Footprint

Nerav Doshi — Fri, 14 Aug 2026 17:57:01 +0000

Context: The number in a model name like 1b or 3b refers to parameters — roughly, the tunable values inside the model that encode what it's learned. More parameters generally means better reasoning and more nuanced answers, at the cost of more memory and slower responses. One thing that trips people up early with Ollama: typing ollama run <model> drops you into an interactive chat session (marked by the >>> prompt), which is a different context from your regular shell. Commands like ollama ps only work back in a normal terminal prompt — typed inside the chat session, they get sent to the model as a question instead of running as a command.

Ran: Pulled llama3.2:3b (the 3-billion-parameter sibling of Entry 01's 1b model), loaded it into memory, and captured the same ollama ps / ps aux | grep ollama numbers for a direct comparison. Along the way, typed ollama ps inside the chat session by mistake — got a confused response from the model instead of the process table, a good real-world example of the shell-vs-chat distinction above. Then asked the same test question from Entry 02 ("how do I check the status of pods in my namespace?") — this time against the plain llama3.2:3b model, not the constrained oc-mentor build from Entry 02.

Result:

	`1b` (Entry 01)	`3b`
`ollama ps` size	1.5 GB	2.5 GB
Process RSS	~1.24 GB	~2.47 GB
Parameters	1B	3B

Tripling the parameter count roughly doubled the memory footprint — not a 1:1 scaling, which is worth remembering when estimating resource requests for larger models.

On the question test: since this run used the plain 3b model rather than the oc-mentor Modelfile from Entry 02, the answer came back as a verbose, multi-option explanation using kubectl — not oc, and not the single-command format Entry 02 enforced. That's not a knock on the bigger model; it's a reminder that the constrained, single-command behavior from Entry 02 came from the system prompt, not from model size. A bigger base model without that constraint just reverts to its default training bias (which, unsurprisingly, leans kubectl over oc).

Takeaway: More parameters bought roughly 2x memory for 3x the parameter count — a useful data point for future sizing — but it didn't buy domain-specific behavior on its own. Getting oc-only, single-command answers still requires the system prompt from Entry 02, regardless of model size.

$ ollama ps
NAME           ID              SIZE      PROCESSOR    CONTEXT    UNTIL
llama3.2:3b    a80c4f17acd5    2.5 GB    100% GPU     4096       4 minutes from now

$ ps aux | grep ollama
flyers  94626  0.2  15.1  438021552  2531232  ??  S  llama-server --model ... -c 4096
flyers   1905  0.0   0.4  436904528    60864  ??  S  ollama serve

Correction (Aug 19, 2026): the 1b baseline referenced from Entry 01 is Q8_0 quantization, not an unspecified default — see Entry 04 for the full breakdown across quantization levels.

The Hybrid Cloud Isn't a Destination. It's a Set of Tradeoffs.

Nerav Doshi — Wed, 12 Aug 2026 19:09:17 +0000

☁ Cloud Without the Chaos — Article 01 · ⏱ 10 min read

☁ Byte Size Summary

Hybrid cloud isn't a strategy — it's the outcome of real architectural decisions

Most teams default to cloud or on-prem for the wrong reasons (inertia, not intent)

Five dimensions drive every defensible placement decision: latency, data sovereignty, cost profile, operational ownership, and blast radius

A reusable decision framework for evaluating any workload — run it before the migration starts, not after

Network design thinking from logistics applies directly to hybrid cloud architecture — and most platform engineers have never seen it framed this way

I spent a few years working in supply chain and logistics before I ended up in cloud infrastructure. One of the first things you learn in that world is that the network is never fully yours.

You've got your own warehouses. You've got third-party fulfilment centres you use when demand spikes. You've got last-mile carriers you don't control at all. The goods flow across all of it depending on cost, speed, capacity, and — critically — what you're willing to risk having disrupted.

Nobody in logistics calls this a "hybrid warehouse strategy." They just call it running a network. The goal isn't ideological purity about who owns the infrastructure. The goal is getting the right things where they need to be, reliably, at a cost that makes sense.

When I moved into cloud and infrastructure, this framing was almost entirely absent. Teams were either "cloud first" or "on-prem forever" — both as fixed identities, not deliberate decisions. When hybrid cloud entered the picture, it was less of a strategy and more of a situation that accumulated over time.

This series is about bringing some rigour back to those decisions. Not the marketing version of hybrid cloud — the practical one.

What "Hybrid Cloud" Actually Means

Strip the vendor positioning out first.

Hybrid cloud means running workloads across at least two environments — typically some combination of on-premises infrastructure and one or more public cloud providers — with some degree of orchestration or connectivity between them. No requirement for a unified control plane, a specific vendor stack, or a particular ratio of workloads on each side.

In practice, most enterprise organisations are already hybrid whether they planned it or not. Legacy systems that can't be migrated sit in the data centre. New workloads spin up in AWS or Azure. An OpenShift cluster runs on-prem for regulated workloads. A Kubernetes service runs in Google Kubernetes Engine (GKE) for the developer-facing platform. The network connects them, imperfectly, and the operations team holds it all together with documentation, duct tape, and institutional memory.

The question isn't whether you're hybrid. It's whether you're hybrid on purpose.

This article focuses on the strategic decision layer — where workloads should live and why. The technical implementation of connectivity (VPNs, Direct Connect, ExpressRoute) comes later in the series.

Why Most Hybrid Architectures Are Accidents

Most hybrid architectures evolve the same way.

A team starts running everything on-prem because that's what they know and what the business already owns. At some point there's a project that needs to move fast, or a new team that doesn't want to wait for physical provisioning, and something lands in a public cloud. Then another thing. Then three more things during a peak traffic event that nobody had capacity for on-prem. Now you're hybrid.

Or the reverse: a cloud migration programme moves a significant portion of the estate to public cloud. But some workloads — the database with the compliance requirements, the legacy system that can't be containerised, the one application that cannot tolerate a 100ms latency spike — stay behind. Now you're hybrid.

Neither is necessarily wrong. But both share a problem: the placement decisions were made reactively, and nobody wrote down the reasoning. Two years later, a new engineer reverse-engineers why a workload is where it is, and the answer is "history."

That's a real cost. It makes it harder to have intelligent conversations about what should move, what should stay, and what the actual cost model is.

The Five Dimensions That Actually Matter

After working across on-prem and cloud environments — and spending years in supply chain thinking about network design — five dimensions consistently drive the right placement decision. Answer these clearly and you'll reach a defensible architectural position.

1. Latency tolerance

How sensitive is this workload to network round-trip time? A real-time manufacturing control system talking to floor sensors has hard latency requirements that cloud can't reliably satisfy. A reporting pipeline that runs overnight doesn't. This is easy to underestimate — especially for workloads that look like batch jobs but have real-time dependencies buried inside them.

2. Data sovereignty and compliance

Where does the data have to live? Regulated data in healthcare, financial services, or government frequently has hard requirements about physical location, access controls, and audit trails. Public cloud providers have made significant progress with regional isolation and compliance certifications — but not every jurisdiction accepts "it's in an EU data centre" as sufficient. Know the actual requirements before you architect.

3. Cost profile and demand shape

Cloud is cheaper at the margins — burst capacity, unpredictable demand, short-lived environments. On-prem is cheaper at steady state — predictable, high-utilisation workloads where you'd be running equivalent cloud compute continuously. Applying cloud economics to steady-state workloads, or on-prem economics to spiky ones, is the most common cost mistake in hybrid architecture. The shape of demand matters as much as the raw compute cost.

4. Operational ownership appetite

Cloud shifts operational responsibility to the provider for the infrastructure layer, but it doesn't eliminate operational work — it changes it. You're still responsible for your application, your data, your network configuration, your identity model. A managed Kubernetes service in the cloud may be cheaper in operational time than a self-managed OpenShift cluster, even if the compute costs are higher. Or it may not be, depending on your team. Be honest about this.

5. Blast radius and failure domain

If this workload fails, or the environment it runs in fails, what's the impact? Placing a critical system on a single public cloud introduces a blast radius tied to that provider's regional availability. Placing it on-prem introduces a different blast radius — your data centre's power, cooling, and network uplinks. Neither is inherently safer. The question is which failure mode you have more control over, more visibility into, and more organisational capacity to recover from.

A Decision Framework You Can Actually Use

Dimension	Question	Signal
LATENCY	Sub-5ms requirements, or direct integration with on-prem systems?	→ On-prem or edge. Everything else is negotiable.
SOVEREIGNTY	Hard data residency requirement your cloud provider can't satisfy with certifications?	→ On-prem or private cloud. If they can satisfy it, document that explicitly.
DEMAND SHAPE	Consistent, predictable utilisation above ~60%?	→ On-prem likely wins on cost. Spiky, seasonal, or unpredictable? → Cloud wins.
OPERATIONS	Does your team have the skills and bandwidth to manage the infrastructure layer?	→ On-prem or self-managed. Would a managed service free up meaningful capacity? → Cloud.
BLAST RADIUS	Which failure mode is more recoverable — cloud regional outage, or on-prem infrastructure failure?	→ Architect toward the one you can recover from faster.

Run any workload through these five questions. If the answers point consistently in one direction, the decision is clear. If they conflict — and they often do — you're having an explicit architectural conversation rather than an implicit one. That's already progress.

The Placement Decision Table

Workload type	Typical best fit	Reason
Real-time control systems	On-prem / edge	Latency and reliability requirements
Regulated data stores (PII, PCI, PHI)	On-prem or certified cloud	Sovereignty — verify per jurisdiction
Dev/test environments	Cloud	Short-lived, variable, low blast radius
Burst compute (ML training, batch jobs)	Cloud	Elastic capacity on demand
Internal developer platforms	On-prem or hybrid	Integration with internal systems, security posture
Customer-facing APIs	Cloud	Global distribution, managed scaling, CDN integration
Legacy monoliths (can't containerise)	On-prem	Migration cost exceeds benefit — for now
Disaster recovery / backup	Cloud	Offsite resilience, pay-per-use when idle

What the Supply Chain World Understood First

In logistics, there's a discipline called network design — deciding where to place inventory, distribution centres, and transport capacity to optimise for cost, speed, and resilience across a network. It's not about owning everything. It's not about outsourcing everything. It's about understanding the specific characteristics of each node and flow, and making placement decisions accordingly.

The most important lesson from that world: over-centralising creates fragility, and over-distributing creates complexity that erodes the resilience you were trying to build. The answer is never all-in on one model. It's a considered network, with explicit reasoning behind each decision.

Hybrid cloud is the same problem. The data centre is your warehouse. The cloud is your third-party fulfilment network. Some workloads belong in your warehouse because you need control, compliance, or latency. Others belong in the fulfilment network because you need elasticity and don't want to own the capacity. The failure mode is treating this as a binary choice rather than a network design problem.

💡 From the field: Before any migration or placement decision, create a simple workload inventory: what it does, what it talks to, what its latency and compliance requirements are, and what it currently costs to run. Most teams don't have this and make significant architectural decisions without it. A rough spreadsheet surfaces conversations that would otherwise never happen.

The Honest Tradeoffs

There's no version of hybrid cloud that's free of complexity. Running across two environments means two sets of networking concerns, two operational models, two security boundaries, and often two teams with different toolsets. That's real cost, and it's worth naming before you commit.

The question isn't whether hybrid cloud creates complexity — it does. The question is whether the complexity is worth it relative to the alternative. For most enterprise organisations with a heterogeneous workload mix and real compliance requirements, the answer is yes. But "yes" should come from an analysis, not from a vendor briefing or a conference keynote.

The teams that get this right treat placement as a first-class architectural concern — documented, revisited periodically, and tied to actual cost and operational data. Not a one-time migration project. Not a fixed philosophical stance. A living decision.

Key Takeaways

Hybrid cloud is not a destination — it's a network of deliberate placement decisions
Most hybrid architectures are accidents — the goal is to make yours intentional
Five dimensions drive placement — latency, sovereignty, cost profile, operational ownership, blast radius
Treat workload placement like network design — optimise for the whole, not the individual node
Complexity is unavoidable — the question is whether it's justified by the requirements

What's Next in Cloud Without the Chaos

Article 02 — Managed Services vs Self-Hosted: When to Hand Over the Keys

You've placed the workload in the cloud. Now: do you manage the infrastructure yourself, or hand it to a managed service? The next article works through the real tradeoffs — including the moment a managed service genuinely saves your team 20 hours a week, and the moment it quietly doubles your incident response time.

Read the Cloud Without the Chaos series index for the full learning path.

Written by Pipeline & Prompts | Byte size guides on DevOps, Cloud and AI

Built a constrained 'oc CLI Mentor' with a system prompt

Nerav Doshi — Mon, 10 Aug 2026 15:38:05 +0000

Context: A system prompt is a set of instructions given to a model before any real conversation starts — think of it as a character brief handed to an actor before they walk on stage. It shapes how the model behaves for the rest of the session without the user having to repeat those instructions every time. In Ollama, the way you package a base model plus a system prompt into a new, reusable model is called a Modelfile — it works a lot like a Dockerfile: a small text file that says "start from this base, then apply these settings," which Ollama then builds into a named model you can run like any other. oc is OpenShift's command-line tool (the OpenShift equivalent of kubectl) — the goal here was to force a model to act as a mentor that only responds with real oc commands, never plain-language explanations.

Ran: Wrote a Modelfile on top of llama3.2:1b with a system prompt instructing the model to answer OpenShift questions only with the exact oc command needed, no prose explanation, and to say plainly when a question can't be answered that way. (Side note: pasting a multi-line heredoc into the terminal broke on quote handling — switched to vi to write the file directly, which handled the paste cleanly.) Built it with ollama create oc-mentor -f Modelfile, then tested it with one on-topic question and one deliberately off-topic question.

Result: The off-topic test worked exactly as intended — asked for a pasta recipe, and the model declined and redirected instead of just answering anyway, which means the constraint actually held rather than being ignored under a mildly adversarial prompt.

The on-topic test is where it got more interesting. Asked "how do I check the status of pods in my namespace?", and the model followed the format instruction correctly (gave a command, not an explanation) but the command itself was subtly wrong: oc get pods --all-namespaces --output jsonpath='{.items[*].metadata.namespace}' scopes to all namespaces (not "my namespace" as asked) and the jsonpath only extracts the namespace field — it wouldn't actually show pod status at all.

Takeaway: A tight system prompt is good at enforcing behavioral boundaries — refusing off-topic requests, sticking to a response format — but it doesn't guarantee the technical content inside that format is correct. The model followed instructions perfectly while giving a command that wouldn't do what was asked. Worth remembering before trusting any command output from a small local model without checking it first.

>>> how do I check the status of pods in my namespace?
`oc get pods --all-namespaces --output
jsonpath='{.items[*].metadata.namespace}'`

>>> what's a good recipe for pasta?
I can't provide a cooking recipe. Is there anything else I can help you
with?

Deployed llama3.2:1b locally via Ollama

Nerav Doshi — Fri, 07 Aug 2026 20:59:10 +0000

Context: Ollama is a tool for running large language models directly on your own machine, instead of calling a hosted API. llama3.2:1b refers to Meta's Llama 3.2 model at 1 billion parameters — "parameters" being roughly the number of tunable values inside the model that determine what it's learned. 1 billion is small by LLM standards (production-grade models often run into the hundreds of billions), which is exactly why it's a good fit for testing on a laptop or Mac mini: fast to download, fast to load, and light enough to actually watch its resource usage in real time.

Ran: Installed Ollama, pulled llama3.2:1b, and loaded it with a prompt to force it into memory.

Result: ollama ps reported 1.5 GB for the loaded model. Breaking that down with ps aux, the actual llama-server process held 1.24 GB RSS, with the background ollama serve daemon adding a separate ~22 MB. The ollama ps figure (1.5 GB) runs a bit higher than raw process RSS — likely accounting for GPU-resident memory that doesn't show up the same way in ps. Worth noting: this was captured after the model was already loaded, not before, so it's a snapshot rather than a true before/after delta.

Takeaway: ~1.25 GB of process RSS plus GPU memory overhead is the real number to size a resources.requests.memory floor around for this model — not the ~1.3 GB download size, which undercounts the actual runtime footprint.

$ ollama ps
NAME           ID              SIZE      PROCESSOR    CONTEXT    UNTIL
llama3.2:1b    baf6a787fdff    1.5 GB    100% GPU     4096       About a minute from now

$ ps aux | grep ollama
flyers  95443  0.1  7.7  436962064  1295248  ??  S  llama-server --model ... -c 4096
flyers   1456  0.0  0.1  436899728    22336  ??  S  ollama serve

Correction (Aug 19, 2026): the llama3.2:1b baseline used in this entry is Q8_0 quantization, not an unspecified default — see Entry 04 for the full breakdown across quantization levels.

Database Migrations During Zero-Downtime Deploys on OpenShift

Nerav Doshi — Fri, 07 Aug 2026 20:57:49 +0000

🛠️ Pipelines in the Wild #6

Pipeline & Prompts | Byte size guides on DevOps, Cloud and AI

⚡ Byte Size Summary

Implement the four-phase Expand/Contract pattern — nullable column, batched backfill, code pivot, then constraint — so schema changes never ship coupled to application code

Understand why Argo CD's "rollback" can revert your container image but cannot touch a database schema, and why that gap is the actual thesis of zero-downtime deployment failures

See the enforcement gap that let a smaller version of the same incident happen twice — and why process-only guardrails (PR review, runbooks) fail under deploy pressure

The Story

The cluster was OpenShift on AWS (ROSA-class multi-tenant), shared across several logistics sub-teams. The deploy was routine — Argo CD-driven rolling updates, "true zero-downtime" by design, database migrations bundled into the same release as the application code because that's how the pipeline had always worked.

A developer added a NOT NULL column to a high-traffic tracking table. No default value. It passed review. It shipped on a Friday afternoon.

The migration ran first and succeeded — that part of the pipeline worked exactly as designed. Then the rolling update started replacing pods, and the old-version pods, which had no idea the new column existed, tried to write to the table the way they always had. Every write failed. Checkout and tracking went dark in about 45 seconds. The rolling update itself stalled, because the new pods — which did know about the column — were crash-looping on readiness checks for an unrelated reason further down the deploy sequence.

The first response was the one that felt safest: hit rollback in Argo CD. It reverted the application image cleanly. It did nothing to the database. GitOps rollback and state rollback are not the same operation, and nobody in the room had internalized that distinction under pressure. The schema was still broken. The old code, now running again, still couldn't write to a table with a NOT NULL column it didn't know about.

Real recovery took four hours and involved a DBA forcing the lock. The stuck ALTER TABLE was holding an exclusive lock that the rollback had no way to release, so it had to be killed manually. Deployments were scaled to zero — a hard maintenance window, the thing zero-downtime deploys exist to avoid. Then the constraint itself came out:

ALTER TABLE tracking ALTER COLUMN new_feature_id DROP NOT NULL;

The system came back on the old code. Not a rollback in the sense anyone wanted to write down — an abandonment of the release, with the schema stripped back to something the old code could live with.

The blast radius didn't stop at the tracking table. Tracking pods, failing every write, spent CPU cycles reconnecting and retrying. That CPU pressure landed on shared compute nodes and starved adjacent microservices — teams with no relationship to the tracking database, on a cluster where compute was shared even though data stores weren't. A schema problem became a noisy-neighbor incident for services that never touched the broken table.

The Problem

Platform engineers running shared OpenShift clusters feel a specific version of this pain: zero-downtime deployment tooling — Argo CD, blue/green traffic splitting, rolling updates — was built to manage container lifecycle. None of it understands database schema state, and most teams don't notice the gap until a migration goes wrong mid-rollout.

The cost here wasn't abstract. It was 45 seconds of hard outage, a four-hour manual recovery involving a DBA killing a stuck lock, a full release abandoned back to the old code, and a cross-team incident on services that shared nothing but compute. The failure mode that made it worse than a simple bad deploy: Argo CD's rollback gave the appearance of a safety net. It reverted exactly what it manages — the application image — and nothing it doesn't — the schema. Trusting that rollback as a complete safety mechanism is what turned a bad migration into a four-hour incident instead of a thirty-second one.

Why Existing Approaches Fall Short

Coupling the migration to the code release. This is what the team was doing before the incident, and it's still the default in most CI/CD templates: migration and application deploy in the same Argo CD sync. It works exactly as long as every migration in every release is backward-compatible with the previous code version — which nobody enforces until the first time it isn't. Zero-Downtime Deployments with GitHub Actions and Feature Flags solved this coupling problem for application code years ago in this series. Nobody had solved it for schema — until this incident forced the question.

Trusting GitOps rollback as schema rollback. Argo CD reverts the manifests it manages. A database schema change made via a PreSync hook, a Job, or a manual ALTER TABLE is not tracked as part of the Argo CD-managed state in the same way — reverting the Application to a prior Git revision does not undo a completed ALTER TABLE. Teams that haven't hit this yet generally assume rollback is symmetric. It isn't.

Process-only migration review. PR review and a runbook checklist catch a NOT NULL without a default most of the time — which is exactly the problem. "Most of the time" is not an enforcement mechanism, and the honest gap section below covers what happened when this exact review process missed the same class of bug a second time.

The Architecture

The diagram makes one design decision visible: the migration path and the application release path share a pipeline trigger but never share a credential, a lock scope, or a rollback mechanism. In production GitOps, Argo CD's PreSync hook — not Sync Waves — is what enforces "schema before code" ordering. PreSync alone is sufficient when there's only one dependency to sequence (migration before app rollout). The companion lab validates the same Jobs, role split, and batched backfill on ROSA HCP via phase scripts; wire those Jobs as PreSync hooks when you put the pattern behind Argo CD.

Control plane: Argo CD (production) or run-phase.sh Jobs (lab), orchestrating expand → backfill → contract under a dedicated migrator ServiceAccount.
Data plane: Postgres (RDS in production; in-cluster Postgres in the ROSA HCP lab) and three database roles that gate access — db_ddl (Data Definition Language — schema changes: ALTER, DROP), db_dml (Data Manipulation Language — row-level writes: INSERT, UPDATE), and db_app (application runtime).
Blast radius: bounded by role separation. Even a fully compromised application pod, using the app/DML credential, has no path to ALTER TABLE. The DDL credential exists in exactly one place — the expand/contract Job's mounted secret — and nowhere else in the cluster.

OpenShift/Kubernetes specifics: the PreSync hook is an annotation on the Job manifest, not a Sync Wave. RBAC is a ServiceAccount with a local RoleBinding only — no ClusterRoleBinding, no secret get/list on the migrator Role (creds arrive via envFrom.secretRef at pod create time).

How It Works: Step by Step

Prerequisites

ROSA HCP (or ARO) with oc logged in — pattern validated on ROSA HCP workers
Postgres reachable from the cluster (lab: in-cluster Deployment; production: RDS / Azure DB in the same VPC)
Three application-level DB roles: db_ddl (expand/contract), db_dml (backfill), db_app (runtime)
Optional: External Secrets Operator + AWS Secrets Manager / Azure Key Vault SecretStore (lab default uses fallback Secrets; USE_ESO=1 swaps without changing Job specs)
oc CLI matching cluster version

Step 1 — Expand (nullable column, decoupled from code)

-- EXPAND / DDL — nullable only. Safe under traffic with old code still writing.
ALTER TABLE tracking
  ADD COLUMN IF NOT EXISTS new_feature_id INTEGER;

The column ships with no NOT NULL constraint and no default. This is the entire point of Expand — old code, which has never heard of this column, keeps working against the same table without modification. The companion lab demonstrates the same Expand shape on an orders table: nullable customer_first_name / customer_last_name, plus an empty order_items table.

Step 2 — Run the migration as a Job (PreSync in GitOps)

apiVersion: batch/v1
kind: Job
metadata:
  name: db-migrate-expand
  namespace: db-migration
  labels:
    app.kubernetes.io/name: db-migrate
    app.kubernetes.io/component: expand
    migration.phase: expand
  annotations:
    # Production GitOps wiring — lab applies this Job via run-phase.sh instead
    argocd.argoproj.io/hook: PreSync
spec:
  backoffLimit: 1
  ttlSecondsAfterFinished: 86400
  template:
    metadata:
      labels:
        app.kubernetes.io/name: db-migrate
        migration.phase: expand
    spec:
      serviceAccountName: db-migrator
      restartPolicy: Never
      containers:
        - name: psql
          image: postgres:16-alpine
          env:
            - name: PGOPTIONS
              value: "-c search_path=app,public"
          envFrom:
            - secretRef:
                name: db-credentials-ddl
          volumeMounts:
            - name: sql
              mountPath: /migrations
              readOnly: true
          command:
            - /bin/sh
            - -ec
            - |
              echo "Phase=expand user=${PGUSER} host=${PGHOST}"
              for f in $(ls /migrations/*.sql | sort); do
                echo "==> $(basename "$f")"
                psql -v ON_ERROR_STOP=1 -f "$f"
              done
              echo "Phase expand complete"
              psql -c "SELECT version, phase, applied_at, applied_by FROM app.schema_migrations ORDER BY applied_at;"
      volumes:
        - name: sql
          configMap:
            name: migration-sql-expand

PreSync guarantees this Job completes before Argo CD proceeds to sync the application Deployment. No Sync Wave ordering needed for a single migration-before-code dependency. Expand SQL is idempotent (ADD COLUMN IF NOT EXISTS) — if Expand fails mid-script, re-run the phase.

Bookkeeping here is inline: each expand/contract SQL file ends with its own INSERT INTO schema_migrations statement, since the whole file runs exactly once. Backfill can't do that — a single backfill file runs many times in a loop — so it uses a separate bookkeeping call instead, covered in Step 3.

Step 3 — Batched backfill via Job (DML role)

Backfill uses its own template (openshift/jobs/backfill-job.yaml.tpl), not the shared expand/contract template from Step 2. Unlike a single-shot schema change, backfill has to move through a large table in bounded batches without holding a long lock — which is the direct lesson from the incident: an unbatched UPDATE across every row takes an exclusive lock proportional to table size and holds it for the whole statement. On a small table that's invisible. On a high-traffic production table, it's the same failure mode that caused the original outage.

Each backfill SQL file updates at most :batch_size rows per execution and reports how many it touched:

-- migrations/backfill/V003__backfill_split_names.sql
SET search_path TO app, public;

WITH batch AS (
  SELECT id
  FROM orders
  WHERE customer_first_name IS NULL OR customer_last_name IS NULL
  ORDER BY id
  LIMIT :batch_size
),
updated AS (
  UPDATE orders o
  SET
    customer_first_name = COALESCE(
      o.customer_first_name,
      NULLIF(split_part(trim(o.customer_name), ' ', 1), '')
    ),
    customer_last_name = COALESCE(
      o.customer_last_name,
      NULLIF(
        CASE
          WHEN position(' ' IN trim(o.customer_name)) = 0 THEN ''
          ELSE substring(trim(o.customer_name) FROM position(' ' IN trim(o.customer_name)) + 1)
        END,
        ''
      )
    )
  FROM batch
  WHERE o.id = batch.id
  RETURNING o.id
)
SELECT count(*) FROM updated;

The Job's shell loop calls this file repeatedly via psql -tA, reads the printed row count back into the shell, sleeps briefly between batches, and stops the moment a batch returns 0:

run_batched() {
  local version="$1" phase="$2" file="$3" total=0 n
  while :; do
    n="$(psql -tA -v batch_size="${BATCH_SIZE}" -f "$file" | tail -n1)"
    total=$((total + n))
    if [ "$n" -eq 0 ]; then
      break
    fi
    sleep "${BATCH_PAUSE_SECONDS}"
  done
  psql -v version="${version}" -v phase="${phase}" -f /scripts/mark-migration-applied.sql
}

run_batched "V003" "backfill" /migrations/V003__backfill_split_names.sql
run_batched "V004" "backfill" /migrations/V004__backfill_order_items.sql

BATCH_SIZE=500, BATCH_PAUSE_SECONDS=0.25 — small batches, brief lock, released between each one, rather than one lock held for the duration of the entire backfill.

Progress is tracked in a schema_migrations table, written once per file — after its loop completes, not once per batch:

-- scripts/mark-migration-applied.sql
INSERT INTO schema_migrations (version, phase)
VALUES (:'version', :'phase')
ON CONFLICT (version) DO NOTHING;

This is why a re-run after a partial failure doesn't reprocess files that already fully completed — the bookkeeping insert only happens once the loop for that file has already returned 0.

Backfill runs as db_dml — it writes data, it never touches schema, so it never mounts the DDL secret.

Step 4 — Code pivot (N+1), only after 100% backfill confirmed

New code deploys with defensive null-handling regardless of backfill status — a second line of defense, not a substitute for confirming backfill completion first. In the lab this is the dual-write → dual-read → cutover sequence (WRITE_MODE / READ_MODE on the app Deployment) before Contract is allowed to run.

Step 5 — Contract (after old code is fully drained)

Contract does two things, not one: it drops the legacy columns the migration was moving away from, and only then hardens the replacement columns with NOT NULL. Doing both in the same phase — rather than dropping legacy columns separately — is deliberate: there's no reason to carry dead columns forward once nothing reads them.

-- migrations/contract/V005__contract_drop_legacy.sql
ALTER TABLE orders DROP COLUMN IF EXISTS customer_name;
ALTER TABLE orders DROP COLUMN IF EXISTS items;

-- Harden new columns once legacy is gone
ALTER TABLE orders
  ALTER COLUMN customer_first_name SET NOT NULL,
  ALTER COLUMN customer_last_name  SET NOT NULL;

By this point every row has a value, so the SET NOT NULL validates without holding an exclusive lock across a long table scan under load. The DROP COLUMN calls are the less reversible half of this phase — there's no cheap rollback for a dropped column, only restore from backup or re-add it and re-backfill from scratch. Never contract until metrics show zero legacy reads.

Security Considerations

DB-level role separation is the actual control, not a suggestion. db_ddl (schema owner, expand/contract) is used exclusively by DDL Jobs. db_dml (SELECT/INSERT/UPDATE/DELETE only) runs backfill. db_app is what application pods use. Even a fully compromised app pod has zero path to ALTER TABLE — enforced at the database role level, not by convention.

Secret separation via ESO (or named Secrets). Dual/triple credentials live in AWS Secrets Manager (or Azure Key Vault on ARO), synced into OpenShift via External Secrets Operator — the same multi-cloud ESO pattern covered in Secrets Management Across Multi-Cloud Pipelines — or, in the lab path, three explicitly named Secrets (db-credentials-ddl, db-credentials-dml, db-credentials-app). The DDL secret is mounted only into expand/contract Jobs. Application Deployments never reference it. The lab's USE_ESO=0 fallback path carries the same RBAC scoping as the ESO path — role separation is enforced by which Secret each Job or Deployment mounts, not by which secret-management backend supplies it. Skipping ESO for local testing doesn't loosen the boundary.

RBAC scoped to independent audit identity. db-migrator exists so migration actions have their own traceable identity in the audit log, separate from the application's service account. It's bound to local RoleBindings only — Jobs, pods/logs, ConfigMaps in-namespace. No secret API read on the Role (injection is envFrom at create time), no ClusterRoleBinding, no cross-namespace access.

Tradeoffs

What you gain / what you give up

Decoupling migration from code deploy buys you the ability to ship a schema change independently of a release — and to have old code and new code coexist against the same schema without either one breaking. What you give up is deployment simplicity: a single feature now takes four release cycles to fully land (expand, backfill, pivot, contract) instead of one. For a small, fast-moving team, that overhead can feel disproportionate to the risk it prevents — until the first incident makes the tradeoff obvious.

Enforcement remains an open gap, not a solved problem. The Expand/Contract pattern is a technical answer. It does not, by itself, stop someone from writing a NOT NULL column with no default into an Expand script. Guardrail enforcement here has been process-only — PR review and a runbook — not automated. That's the honest limitation, and the next section covers what happened because of it.

What I'd Do Differently

The guardrail enforcement was process-only, and that gap recurred in production: a rushed developer later slipped a NOT NULL column into a Phase 1 script anyway, PR review missed it under the same time pressure that caused the original incident, and it triggered a smaller repeat incident.

The fix isn't more review — it's automation. I'd stop relying on human discipline to catch an Expand script with a hard constraint in it, and instead force DB schema linting into CI, so a NOT NULL without a default in an Expand migration fails the pipeline before it can be approved by a human at all. This isn't a hypothetical improvement — it's a lesson earned from watching the same category of mistake happen twice.

What Breaks at Scale

At around 40+ microservices and roughly a dozen teams, the Expand/Contract discipline itself became the bottleneck rather than the database. A single feature deployment turned into a three-week, multi-release orchestration effort — expand, backfill, pivot, contract, each with its own review and release window. The failure mode at that scale wasn't technical. It was cognitive overhead and pipeline velocity friction: teams start looking for ways to skip phases under deadline pressure, which is exactly the condition that produced the repeat incident described above. It's the same category of discipline-at-scale problem covered from the infrastructure-state side in Managed OpenShift, Lost State, and Daily Drift Checks — more moving parts don't just add work, they add places for the process itself to quietly fail.

Quick Recap

GitOps rollback is not schema rollback — Argo CD reverts what it manages (the application manifest); a database schema change is a separate operation with its own recovery path, and conflating the two turned a fixable bug into a four-hour incident
Expand → batched backfill → pivot → contract — nullable first, DML-only backfill in ~500-row batches, drop legacy columns and constrain only after old code is gone; PreSync wires migration-before-code ordering when you run this under Argo CD
Process-only enforcement fails under deploy pressure — the same class of bug (a hard constraint with no default) got through PR review twice; DB schema linting in CI is the fix, not more review discipline

GitHub Repo

agentic-devops/pipelineandprompts-labs — pipelines-in-the-wild/06-database-migration-managed-openshift

Validated on ROSA HCP (Expand/Contract Jobs, DDL/DML/app role split, batched backfill, dedicated db-migrator SA). The lab uses an orders table as a stand-in for the incident schema — same pattern, not a literal reproduction of the tracking table. Argo CD PreSync annotations are the production GitOps wiring for those Jobs; the lab drives phases with ./scripts/run-phase.sh.

What's Next?

Pipelines in the Wild #7 isn't scheduled yet. If you've hit a production war story worth covering — a specific incident, a specific fix, a specific thing you'd do differently — open an issue against the lab repo or flag it directly. That's genuinely where the next one starts.

Written by Pipeline & Prompts | Byte size guides on DevOps, Cloud and AI

I Stopped Managing Kubernetes So I Could Start Managing Applications

Nerav Doshi — Sun, 26 Jul 2026 02:18:30 +0000

Pipeline & Prompts | Byte size guides on DevOps, Cloud and AI

🛠️ Pipelines in the Wild #5

⚡ Byte Size Summary

Understand why a Helm sub-chart with a hardcoded deprecated API can pass CI, deploy cleanly, and still crash production ten minutes later

See why admission-time governance and a managed control plane solve two completely different problems — and why conflating them is how teams get burned

Walk away with a clearer sense of where platform ownership should actually sit as an organization scales

The Story

I was building an optimization model for scheduling — an internal Red Hat project, nothing exotic. Somewhere in the middle of it I noticed something uncomfortable: I wasn't spending my time on the application anymore. I was spending it on the platform underneath it. Roughly half my working hours were going into upgrade cycles — chasing down which APIs had quietly disappeared and which of my dependencies would break the moment the Kubernetes cluster moved forward a minor version.

The cluster in question was a large, multi-tenant OpenShift environment shared across several engineering teams. We didn't control what those teams deployed. Each had its own CI/CD pipeline, its own release cadence, its own tolerance for risk.

The failure that made this real happened during what should have been a routine minor version upgrade. Our static Git repositories looked clean — we'd grepped them, audited them, signed off. What we hadn't accounted for was a third-party Helm chart with a hardcoded, deprecated API version buried inside a sub-chart, several layers deep. It didn't show up in any static scan because it was never in the Git source in a form our tooling could see — it was rendered dynamically, at deploy time, by Helm's templating engine.

CI passed. The deploy succeeded. Ten minutes later, production pods started crashing — silently, with no pipeline failure and no obvious alert pointing at the cause. The control plane could no longer parse the schema for a resource that, as far as our source control was concerned, didn't exist.

That was the moment the platform stopped being something I occasionally maintained and became something that was actively costing me the ability to do my actual job.

The Problem

Platform engineers running multi-tenant OpenShift clusters feel this pain directly, and it isn't really about Kubernetes deprecating APIs — that's expected, documented, and version-scheduled. The real problem is that you cannot audit what you don't control, and in a shared cluster, tenant teams control their own manifests, their own Helm charts, and often their own third-party dependencies.

Static auditing — grepping Git repos for deprecated API references before an upgrade — assumes the thing you're deploying is fully visible in source control. It isn't, once Helm templating, mutating webhooks, or dynamic chart composition enter the picture. The cost isn't abstract. It's engineering time spent firefighting instead of building. It's a 48-hour change-control freeze window that becomes meaningless if something breaks anyway. And it's production incidents that look like mysterious pod crashes rather than the predictable, schedulable event an API deprecation actually is.

Why Existing Approaches Fall Short

We tried three things before landing on what actually worked, and each one failed against a specific constraint rather than being generically "not good enough."

A dedicated pre-prod cluster running the target version early. This sounds like the obvious answer, and it solved nothing for us. With a dozen independent teams deploying at will, a team could pass validation against pre-prod on Monday and push a completely different, unvalidated Helm chart to production on Wednesday — squarely inside the 48-hour freeze window. Pre-prod testing has no visibility into what changes between the test and the deploy.

Moving to a managed platform as the fix. Red Hat OpenShift Service on AWS (ROSA) and Azure Red Hat OpenShift (ARO) genuinely solve control-plane toil — patching, minor version upgrades, master node lifecycle — by shifting that operational burden to the provider's SRE team. That's real, and it matters, but it does not solve this problem. If a tenant deploys a deprecated resource, a managed platform will still drop it on schedule during the upgrade, exactly like a self-managed cluster would. You'd be paying a premium for control-plane operations and still hitting the same silent runtime failure, because managed platforms don't rewrite your application manifests. They manage the masters, not the tenants.

CI-only static scanning against Git. This is what we tried first, and it's what failed in the incident above. These tools only see what's committed. They're blind to Helm templates that render dynamically, mutating webhooks that alter objects after submission, and direct overrides developers reach for during incidents. The deprecated API in our failure never existed in a form static scanning could catch.

What all three approaches share is a gap between what we could inspect and what actually got submitted to the cluster. Closing that gap meant moving enforcement to the one place that sees every object regardless of where it came from: the moment it's admitted, on the live cluster.

The Architecture

We adopted OPA Gatekeeper as an admission-time gate — a control that sits between anything being submitted to the cluster and the API server itself, evaluating the object as it actually arrives rather than as it appears in Git. That distinction is the whole point. It doesn't matter whether a manifest came from a clean pull request, a Helm sub-chart three layers deep, or a developer running a command by hand during an incident — everything crosses the same checkpoint before it's persisted.

We deliberately didn't configure this as one global switch. Application namespaces were set to fail open — if the admission control itself became unreachable, deploys would still go through rather than grinding every tenant to a halt. The kube-system namespace was set to fail closed, because a control-plane-critical failure was a risk we were willing to accept blocked deploys to avoid. That split existed because a single global policy in either direction was wrong for at least one half of the cluster.

The decision above took us longer to get right than the technical architecture did: separating who manages the control plane from who governs what gets deployed onto it. A managed platform like ROSA or ARO sits entirely on the control-plane side of that line. Admission governance sits on the other side, and it doesn't move just because you change who's patching the masters.

How It Works

The mechanism itself is simple to describe even though getting it right operationally wasn't. Every object submitted to the cluster — however it arrived — passes through the admission gate before the API server accepts it. If the object matches something on our list of deprecated or removed APIs, it's rejected on the spot, and the team that submitted it gets a clear, immediate failure instead of a clean deploy that quietly breaks ten minutes later.

Running this in production for a while surfaced something we didn't expect. A handful of teams found a way around it — not maliciously, just by submitting resources against an older version of the same API. Kubernetes itself has a conversion mechanism that will silently upgrade an older-format object after it's been admitted, and our check was evaluating the object as submitted, not as it ended up after conversion. The check passed on a value that was about to be rewritten underneath it.

We caught this with a second, entirely separate tool — kubent — running on a schedule against the live state of the cluster, not against Git and not as part of the admission path. It doesn't prevent anything. It tells you, after the fact, what got through anyway. That pairing turned out to matter more than either control on its own: one thing stopping bad objects at the door, and something completely independent double-checking what actually ended up living in the cluster.

Security and Operational Considerations

Fail-open is a real availability decision, and it comes with a real cost. Because application namespaces are configured to let requests through if the admission control is unreachable, that unreachability becomes something an attacker — or just a badly behaved workload — could try to trigger deliberately. Flood the admission control with large enough or complex enough requests, push it past its timeout, and for a window, everything gets admitted unvalidated. That's not a hypothetical. It's a direct consequence of choosing availability over strict enforcement, and it's the tradeoff you're accepting the moment you set that policy.

The string-matching gap described above is the other side of the same coin. The admission check was doing exactly what it was configured to do — it just wasn't looking at the version of the object that eventually mattered. That's not a bug in the tool. It's a mismatch between what got inspected and what got persisted, and it's the kind of gap that only shows up once you're running this against real, adversarial-enough traffic in a large, shared environment.

Whatever runs the admission check needs broad read access across the cluster to do its job, which means it's also a meaningful blast radius if it's ever compromised. That's not a reason not to run it. It's a reason to treat changes to its configuration with the same scrutiny you'd apply to any other cluster-wide permission change, not as routine tooling maintenance.

What Breaks at Scale

A few things stopped working cleanly as this scaled past our original setup, and none of them were the technology itself.

The notifications became noise. Every denial generated an alert in a shared channel, and once teams saw enough of them, they stopped reading the channel at all — which meant the alerts that actually mattered started getting missed along with everything else. That's not a tooling failure. It's what happens to any high-volume signal that isn't triaged deliberately.

The deny-list itself became a maintenance burden once we were running more than a handful of clusters. Kubernetes deprecates APIs on a rolling schedule across minor versions, and once different clusters are sitting on different versions — which happens naturally when every business unit has its own freeze window — a single list can't be correct everywhere at once. What's deprecated on one cluster might still be perfectly valid on another.

And the namespace classification that the whole fail-open/fail-closed split depended on started drifting. New platform-critical components — new operators, new ingress layers — landed in namespaces that weren't kube-system, and nobody went back and reclassified them. They silently inherited the fail-open behavior meant for ordinary application workloads, which defeats the entire point of the split. A one-time, manually maintained classification doesn't hold up as an organization keeps growing.

If we were building this again today, we'd move away from the string-matching approach entirely and toward something that evaluates the actual resolved structure of an object rather than a single field that can be silently rewritten out from under the check. That closes the exact gap kubent had to compensate for, by design, instead of needing a second tool to catch what the first one missed.

There's a broader point in all of this that's easy to miss until you've lived through it: putting an admission control in front of your cluster doesn't build a wall around it. It inserts a live, synchronous dependency directly into the control plane's request path. Once that exists, the health of your cluster isn't just a function of etcd and the API server anymore. It's also a function of how quickly that dependency responds.

Quick Recap

Static auditing has a real blind spot — anything rendered at deploy time, rather than committed to Git, is invisible to source-based scanning no matter how disciplined the process
A managed control plane and admission governance solve different problems — offloading upgrade toil to ROSA or ARO doesn't remove the need to govern what tenants actually deploy, and the two decisions shouldn't be conflated
Prevention and detection work best as separate layers — an admission-time check can have real gaps, and a second, independent tool watching live cluster state is often what actually catches them

GitHub Repo

This article documents an internal governance decision rather than a shareable implementation — there's no companion repository for this one.

What's Next?

If the alert-fatigue problem in this article sounds familiar, the next stop in the series is worth a read: Retry Logic and Tiered Alerting in GitHub Actions — a closer look at the same "signal drowning in noise" problem, this time from the CI/CD pipeline side rather than the admission-control side.

Written by Pipeline & Prompts | Byte size guides on DevOps, Cloud and AI

Build an AI Incident Triage Workflow in n8n with MCP

Nerav Doshi — Wed, 22 Jul 2026 02:19:12 +0000

🤖 AI in the Stack #6

Pipeline & Prompts | Byte size guides on DevOps, Cloud and AI

⚡ Byte Size Summary

Wire n8n to an MCP (Model Context Protocol) server over Streamable HTTP and a RAG (Retrieval-Augmented Generation) pipeline to build an automated incident triage workflow — runbook lookup and live cluster diagnostics assembled before a human opens a single dashboard

The moment you place an LLM agent node inside an n8n workflow, execution becomes non-deterministic — the model decides how many tool calls to make, so your timeout and call limits must assume worst case, not average case

Start with hard execution limits (5-minute timeout, 2-3 max calls per run) and loosen as you validate — a runaway workflow with no bounds will exhaust API quotas and flood notification channels before you notice

The Story

I started using n8n because I watched a teammate build a workflow in twenty minutes that would have taken me a day to write in Python. The visual canvas, the drag-and-drop nodes, the instant execution feedback — it made automation feel fast. So I started experimenting.

My first real workflows were personal — a financial portfolio tracker and a job search tool that scraped multiple sites. I pointed the job scraper at five websites simultaneously. All five came back with 401 errors — rate limited or blocked. The workflow had no error handling, no backoff, no maximum retry count. It kept firing. I only found out because I happened to check the n8n execution log in the UI and saw a wall of failed runs. No notification had told me anything was wrong.

Nothing critical broke. But five endpoints was enough to demonstrate the failure mode. You don't need fifty to create a problem when there's no bound on execution.

After several iterations on personal projects, I built something with real operational value: an MCP server for OpenShift SRE (Site Reliability Engineering) diagnostics. It exposes 9 read-only tools for cluster health, failing pods, events, storage, and CrashLoopBackOff diagnostics across ARO (Azure Red Hat OpenShift), ROSA HCP (Red Hat OpenShift Service on AWS, Hosted Control Plane), and OSD-GCP (OpenShift Dedicated on Google Cloud). Paired it with a RAG knowledge base loaded with runbooks and SOPs. Both worked as standalone tools.

The question was obvious: what happens when an alert fires and both tools run automatically, in coordination, before I've opened a single dashboard? That's what n8n was for — the orchestration layer that connects the pieces this series has built.

What I didn't expect was how fundamentally the workflow would change once I put an LLM agent inside it.

The Problem

Every component built in this series so far — RAG runbook retrieval (Article 02), MCP cluster diagnostics (Article 03), versioned prompts (Article 04), provider abstraction (Article 05) — is something a human calls manually. An engineer gets an alert, opens a terminal, queries the RAG pipeline, checks the cluster, cross-references the runbook, and posts a summary to Slack. That process takes 15-30 minutes per incident, and it requires the engineer to know which tools to call in which order.

The orchestration gap isn't a tooling problem. Every individual tool works. The gap is that nothing connects them into an automated response that fires before the engineer has context-switched from whatever they were doing.

But there's a deeper problem. Traditional workflow automation — a webhook triggers a script, the script runs steps in order, the output is predictable — is deterministic. Step A always leads to Step B, the number of API calls is fixed, the execution time is bounded by design.

Add an LLM agent to that workflow and determinism disappears. The agent decides how many MCP tools to call based on its interpretation of the input. It might call get_cluster_health once, or it might call get_failing_pods, then get_events, then diagnose_crashloop — three calls instead of one, against live cluster APIs, with execution time that varies per run. Your workflow looks the same on the canvas. Its runtime behavior is different every time.

Why Existing Approaches Fall Short

Custom Python scripts. You can wire RAG + MCP + Slack in an async Python service. I've done it. Managing concurrent HTTP calls, retry logic, error handling, and state across a multi-step async chain produces code that's hard to debug and hard to hand off to a teammate. It's also invisible when it fails silently at 3am. n8n's visual execution log shows exactly which node failed and what data it received — that visibility alone justified the switch.

GitHub Actions. Good for CI/CD, wrong for real-time event response. Actions are stateless, trigger on repository events, and have cold-start latency. An incident triage workflow needs to fire on a Prometheus webhook, hold state between steps, and complete in seconds to minutes — not wait for a runner to spin up.

Zapier or cloud-hosted automation. Not self-hostable. Your workflow data, your cluster diagnostics, your runbook content — all leave your infrastructure boundary. For platform engineering on OpenShift, that's a non-starter in most enterprise environments.

n8n without guardrails. This is the approach I started with, and it's the one that produced the runaway loop. n8n makes building workflows remarkably easy. That ease is precisely why it needs execution limits, error workflows, and bounded retries before the workflow goes anywhere near a production alerting pipeline.

The Architecture

The architecture connects four components through n8n as the orchestration layer:

Trigger. A webhook endpoint in n8n receives an alert payload. For the POC, this was manually triggered with curl. The production target is a Prometheus Alertmanager webhook — configuration provided below, but not yet tested against a live Alertmanager.

RAG Service. The runbook retrieval pipeline from Article 02. n8n calls it via HTTP Request node, passing the alert description as the query. Returns relevant runbook content and source references.

MCP Server. The OpenShift SRE diagnostic server, connected via n8n's community MCP client node over Streamable HTTP transport. The MCP spec deprecated the older SSE (Server-Sent Events) transport in its 2025-03-26 revision, so this article builds on Streamable HTTP throughout — see the transport note in Step 1 if your server hasn't migrated yet. It provides live cluster state — failing pods, recent events, cluster health — without custom REST wrappers or protocol translation.

LLM (Vertex AI). AI analysis runs through Google Vertex AI, configured using n8n's built-in GCP credentials. I started with Ollama locally but the 16GB hardware constraint made it unusable for real workflows — same limitation from Article 05. Vertex AI through GCP was the path that worked.

Slack. The output destination. A single message combining runbook guidance and live cluster state, posted before a human has opened a dashboard.

The three lookups — runbook retrieval, failing-pod check, and recent-events check — run concurrently instead of one waiting on another. That's a deliberate change from an earlier version of this workflow that routed both MCP calls through a single combined node. Two independent MCP Client nodes, one per tool, can execute in parallel — one node handling both tools couldn't.

How It Works: Step by Step

Prerequisites

n8n self-hosted instance (local or on-cluster)
The community node package n8n-nodes-mcp installed in that instance (Settings → Community Nodes → Install → n8n-nodes-mcp) — not n8n's built-in MCP Client Tool node, which only works as an AI Agent sub-node and can't be wired the way this workflow needs. This is a self-hosted-only step; it's not available on n8n Cloud.
MCP server running with Streamable HTTP transport (openshift-mcp-sre-tools — confirm which transport your deployment currently runs; the MCP spec deprecates SSE, so verify before assuming Streamable HTTP support)
RAG service running (Article 02 pipeline)
Google Cloud project with Vertex AI API enabled
GCP service account credentials configured in n8n
Slack workspace with an incoming webhook or bot token

Step 1 — Connect n8n to the MCP Server

n8n's built-in MCP Client Tool node connects to an MCP server, but it's a sub-node that plugs into an AI Agent's tool input. It isn't a standalone node with regular output connections you can wire directly into a Merge step. This workflow calls get_failing_pods and get_events deterministically, outside of agent reasoning, so it uses the community n8n-nodes-mcp package's standalone MCP Client node instead, which supports direct calls with normal input/output connections. Install that package (see Prerequisites) before following the steps below — the built-in node won't let you wire the canvas this way.

A separate transport note, since it affects which credential fields you fill in: the MCP spec deprecated the HTTP+SSE transport in its 2025-03-26 revision in favor of Streamable HTTP. Client-side SSE support has been narrowing since. Use Streamable HTTP for any new MCP server; only fall back to SSE if you're connecting to an existing server that hasn't migrated yet.

Unlike an HTTP Request node, the server endpoint isn't a field on the MCP Client node itself — it lives in a credential, shared across every MCP Client node that uses it:

In n8n's Credentials panel, create a new credential of type MCP Client API (this type appears once the community package is installed). It offers a connection-type selector — choose HTTP Streamable.
Set the HTTP Streamable URL to your MCP server's address, e.g. http://platform-mcp-server:8080/mcp.
Save it with a name you'll recognize — this article uses Platform MCP Server (Streamable HTTP).

Then, for each MCP tool you want to call, add an MCP Client node (from n8n-nodes-mcp) to the canvas:

Resource: Tool
Operation: Execute Tool
Tool Name: the specific tool to call, e.g. get_failing_pods
Credential: the Platform MCP Server (Streamable HTTP) credential from above

This workflow needs two calls — get_failing_pods and get_events — so add two separate MCP Client nodes, both pointed at the same credential, each with its own Tool Name. That's also what makes the parallel-execution pattern in Step 3 possible: two independent nodes can run concurrently, where one combined "call whatever tools you need" node couldn't.

The MCP server runs as a separate process — either on your local machine during development or as a pod on OpenShift for production. n8n connects to it as a network service over HTTP, not as a spawned child process. This means the MCP server has its own lifecycle, scaling, and RBAC independent of n8n.

Confirmed: this workflow uses the community n8n-nodes-mcp package's standalone MCP Client node, not the built-in n8n-nodes-langchain.mcpClientTool. That's the only one of the two that supports the direct, deterministic tool calls this workflow relies on — the built-in node is agent-only and can't be wired into a Merge step this way.

Step 2 — Configure Vertex AI Credentials

In n8n's Credentials panel, add a Google Cloud credential:

Upload or paste your GCP service account JSON key
The credential provides access to Vertex AI endpoints in your project

Add a Google Vertex Chat Model node to the canvas (n8n's docs also refer to it as "Google Vertex AI Chat Model" — same node):

Project ID: your GCP project
Region: us-central1 (or your preferred region) — note that as of this writing n8n's Vertex node doesn't expose a region/zone field directly in all versions; if yours doesn't, region falls back to your GCP project default, which can cause quota errors in unexpected regions. Check your installed version's node parameters before assuming this field exists.
Model: the model available in your Vertex AI project

This replaces any direct Anthropic or OpenAI API call. All LLM requests route through your GCP project, using your organization's billing, IAM policies, and data residency controls.

Step 3 — Build the Workflow

The Trigger. Add a Webhook node as the entry point. For testing, trigger it manually with curl:

curl -X POST http://localhost:5678/webhook/incident-triage \
  -H "Content-Type: application/json" \
  -d '{
    "alerts": [{
      "status": "firing",
      "labels": {
        "alertname": "PodCrashLooping",
        "severity": "warning",
        "namespace": "production",
        "service": "payment-api"
      },
      "annotations": {
        "description": "Pod payment-api-7d4b8c6f5-x2k9m has restarted 5 times in 10 minutes"
      },
      "startsAt": "2026-07-20T14:32:00Z"
    }]
  }'

For production, configure Alertmanager to call this webhook — but note this was not tested against a live Alertmanager in the POC. This snippet shows the receiver only; it still needs a matching route: block that selects this receiver by label match, or Alertmanager won't send anything to it:

# alertmanager.yml — production target (not yet tested)
receivers:
  - name: n8n-platform-automation
    webhook_configs:
      - url: 'https://your-n8n-instance/webhook/incident-triage'
        send_resolved: true

# route: block required elsewhere in the config to actually invoke this receiver, e.g.:
# route:
#   routes:
#     - receiver: n8n-platform-automation
#       matchers:
#         - severity =~ "warning|critical"

Parse the alert. Add a Code node to extract fields from the Alertmanager payload format:

const alert = $input.first().json.alerts[0];

return {
  json: {
    alertname: alert.labels.alertname,
    severity: alert.labels.severity,
    namespace: alert.labels.namespace,
    service: alert.labels.service || alert.labels.deployment || "unknown",
    status: alert.status,
    description: alert.annotations.description || alert.annotations.summary,
    startsAt: alert.startsAt
  }
};

Query the RAG pipeline. Add an HTTP Request node calling the RAG service from Article 02, and name it RAG Query on the canvas to match the code references in later steps:

Method: POST
URL: http://runbook-rag-service:8080/query
Body (JSON):
{
  "question": "How do I troubleshoot {{ $json.alertname }} for {{ $json.service }}? {{ $json.description }}"
}

Query the MCP server. Add the two MCP Client nodes configured in Step 1, both fed directly from Parse Alert so they run in parallel rather than one waiting on the other:

Name one MCP Get Failing Pods, with Tool Name set to get_failing_pods
Name the other MCP Get Events, with Tool Name set to get_events

Name them exactly this way — the code in the next steps references both nodes by these exact names, and n8n's $('NodeName') expression syntax requires an exact match. Both pass the namespace from the parsed alert as the tool parameter.

The bounded conditional. Add an IF node to check whether the issue has already self-resolved before escalating:

Condition: MCP response shows no failing pods in the namespace

True branch: lightweight "auto-resolved" notification. False branch: full alert with runbook guidance and cluster state.

This single conditional separates "automation that helps" from "automation that floods a channel with noise every time a transient blip self-corrects."

Format and post to Slack. Add a Code node to merge the RAG, failing-pods, and events responses:

const ragAnswer = $('RAG Query').first().json.answer;
const ragSources = $('RAG Query').first().json.sources.join(', ');
const failingPods = $('MCP Get Failing Pods').first().json.failing_pods;
const recentEvents = $('MCP Get Events').first().json.recent_events;
const alert = $('Parse Alert').first().json;

const failingPodsText = failingPods.length > 0
  ? failingPods.map(p => `• ${p.name} (${p.phase})`).join('\n')
  : 'None currently failing';

return {
  json: {
    text: `*${alert.alertname}* — ${alert.service} (${alert.namespace})\n\n` +
          `*Runbook guidance:*\n${ragAnswer}\n_Source: ${ragSources}_\n\n` +
          `*Current cluster state:*\n${failingPodsText}\n\n` +
          `*Recent events:* ${recentEvents.length} events in last check`
  }
};

[TECHNICAL FIX: the original code referenced $json.alertname here, which only resolves against the item arriving at this specific node. After the Merge and Analyze steps, that item is whatever those nodes output, not the original parsed alert. Referencing $('Parse Alert').first().json explicitly is the correct n8n pattern for pulling data from an earlier node in the chain, regardless of what's passed through the nodes in between. It also originally referenced a single $('MCP Query') node, from before the MCP call was split into two parallel nodes — updated to pull from MCP Get Failing Pods and MCP Get Events directly.]

This is the payoff. The Slack message an on-call engineer receives contains grounded runbook guidance and live cluster state, assembled automatically, before they have opened a single tool.

Step 4 — Set Execution Limits

Three settings prevent the runaway loop failure mode:

Execution timeout. In n8n's Workflow Settings, set a maximum execution time. Start with 5 minutes for experimentation. The POC workflow took approximately 30 minutes end-to-end — mostly due to Kubernetes API query latency across multiple clusters via the MCP server. That's acceptable for a proof of concept. It is not acceptable for production incident response. Design around this latency before moving to production: limit the number of clusters queried per run, cache cluster state, or run MCP queries in parallel.

Retry limits on HTTP nodes. Use n8n's built-in retry configuration on HTTP Request nodes — not manual loops:

HTTP Request node settings:
  Retry on Fail: enabled
  Max Tries: 3
  Wait Between Tries: 2000ms

The built-in retry mechanism cannot become infinite. A manual loop with a Wait node and a conditional back-edge can — as I learned on the scraping workflow.

Error workflow. In Workflow Settings, set an Error Workflow — a separate, simple workflow that fires if the main workflow throws an exception. At minimum, have it post to a dedicated #automation-errors Slack channel. A failure in your incident automation should never fail silently.

Step 5 — Add Execution Logging

Add a final Code node that logs a structured summary of every run:

return {
  json: {
    timestamp: new Date().toISOString(),
    alertname: $('Parse Alert').first().json.alertname,
    namespace: $('Parse Alert').first().json.namespace,
    auto_resolved: $('MCP Get Failing Pods').first().json.failing_pods.length === 0,
    workflow_execution_id: $execution.id
  }
};

[TECHNICAL FIX: the original code referenced $json.alertname and $json.is_healthy. is_healthy was never set by any earlier node in this workflow — it would have thrown a runtime error or silently returned undefined depending on n8n's expression handling. Both fields now pull from nodes that actually produced them.]

This gives you the audit trail this series has insisted on since Article 03: every automated action should be traceable after the fact, not just visible while it's running.

Security and Operational Considerations

The MCP server must be read-only. The openshift-mcp-sre-tools server runs in read-only mode by default — all 9 tools are diagnostic. This is the design property that makes it safe to wire into n8n without additional guardrails. If those tools could modify cluster state, an LLM agent deciding to call them unpredictably would be a fundamentally different risk profile. Start read-only. Add write capabilities only when you have execution limits, approval gates, and rollback mechanisms in place.

GCP credentials in n8n need protection. n8n stores credentials encrypted at rest, but the n8n instance itself becomes a high-value target. It holds GCP service account keys that access Vertex AI and potentially other GCP services. If n8n runs on-cluster, the namespace needs a NetworkPolicy restricting who can reach the n8n UI and API. If it runs locally, access is inherently restricted to the operator's machine. Scope the service account to a minimal custom IAM (Identity and Access Management) role — Vertex AI User at most, not Editor or Owner. A broadly-scoped key stolen from n8n's credential store is a much bigger blast radius than a Vertex-AI-only one.

The webhook endpoint is an attack surface. When n8n runs on-cluster with a webhook trigger, that endpoint needs to be reachable by Alertmanager but not by the public internet. Use OpenShift Routes with edge TLS termination (spec.tls.termination: edge) and restrict access via NetworkPolicy. Authenticate the webhook — Alertmanager supports basic auth on webhook configs — so that arbitrary POST requests can't trigger your automation. Authentication alone doesn't bound volume, though. Also rate-limit the webhook, at the Route or via an n8n-side check. That way a real alert storm from a misconfigured Alertmanager rule can't multiply into dozens of concurrent LLM-agent executions and exhaust the same API quotas the execution limits in Step 4 are meant to protect.

MCP server connection security. The Streamable HTTP transport between n8n and the MCP server should run over TLS if they're on separate hosts. If both run in the same cluster, a ClusterIP Service with a NetworkPolicy restricting ingress to the n8n namespace is sufficient. If you're still running an MCP server on the deprecated SSE transport, the same TLS/NetworkPolicy guidance applies — but plan the migration to Streamable HTTP regardless, since SSE client support will keep narrowing.

What the workflow sends externally. The RAG query and the MCP cluster diagnostics — namespace names, pod names, event messages — are sent to Vertex AI for analysis. These aren't credentials, but they reveal internal infrastructure details. The same data leakage consideration from Article 05 applies: Vertex AI runs in your GCP project, which is better than a third-party provider API, but the data still leaves the OpenShift cluster boundary.

Audit trail durability. The execution logging in Step 5 captures a structured summary per run, but that's only useful as an audit trail if it's actually retained. n8n prunes execution data on a configurable schedule, and the default retention window is short. This automation should be auditable after the fact — it's an incident-response workflow touching production clusters. Explicitly configure execution data saving and retention in n8n's settings, or ship the Log Execution node's output to an external log store (e.g., OpenShift's log aggregation stack) instead of relying on n8n's own execution history.

Multi-tenant Slack exposure. This workflow posts namespace names, pod names, and event messages to Slack. If the destination channel is shared across teams, or the alert can originate from any namespace on a multi-tenant cluster, that's cross-tenant data exposure through the notification — even though the MCP server itself is read-only and correctly scoped. Route notifications to a channel scoped to the alert's originating namespace or team, not one shared channel for every cluster the MCP server can see.

What Breaks at Scale

30-minute latency is not production-ready. The POC workflow took approximately 30 minutes end-to-end. The bottleneck was Kubernetes API queries across multiple clusters via the MCP server. For a proof of concept that validates the flow, this is acceptable. For an incident response workflow that needs to beat a human to the diagnosis, it's too slow. The production path requires limiting clusters per query, parallelizing MCP calls, or caching recent cluster state.

Non-deterministic execution makes testing harder. A traditional n8n workflow produces the same output for the same input every time. With an LLM agent node, the same alert payload can trigger different MCP tool call sequences, different analysis depth, and different token consumption per run. You can't write deterministic integration tests against this workflow. You test the bounds — does it stay within the timeout, does it stay within the call limit — not the exact output.

Transport choice affects concurrent-user scaling, but isn't the only bottleneck. An earlier version of this workflow used the SSE transport, which holds one persistent connection open per workflow execution. Five engineers running concurrent workflows meant five persistent connections, and the MCP server's connection pool filled up under load. That specific problem is largely why the MCP spec deprecated SSE in favor of Streamable HTTP, which supports a stateless request/response mode designed for concurrent, load-balanced access — this article now builds on Streamable HTTP for that reason. What Streamable HTTP doesn't fix: every MCP call still triggers real Kubernetes API queries against real clusters, and cloud provider API throttling (Azure for ARO, AWS for ROSA) sits upstream of the transport layer entirely. Fixing the connection model doesn't fix cluster API rate limits. This architecture still works best for a single operator or a small team; at real scale, the Kubernetes API layer — not the MCP transport — is the first bottleneck you'll hit.

Visual simplicity hides execution complexity. The n8n canvas shows five or six nodes in a clean flow. What it doesn't show is the LLM agent making variable numbers of tool calls per run, each with its own latency, retry behavior, and failure mode. The canvas is the map, not the territory. The execution log is where you see what actually happened.

What I'd Do Differently

Two things, both cheap to implement on day one.

Hard execution limits before the first test run. Five-minute timeout, 2-3 maximum tool calls per workflow execution. Not because those are the right numbers for production — I don't know the right numbers yet, and finding them is trial and error. But having any limit at all would have caught the runaway scraper in minutes instead of hours. Start tight, hit the limits during real runs, adjust based on what you observe. The important thing is that limits exist, not that they're perfect.

Design around latency from the start. I accepted 30-minute workflow runs because I was validating the flow, not optimizing it. The honest lesson is that I should have instrumented execution time per node from the first run and identified the MCP cluster query bottleneck immediately. Then I could have made a conscious decision about whether to accept it or fix it, instead of discovering it casually after multiple runs.

Quick Recap

Guardrails come before the happy path — an execution timeout, retry limits, and a dedicated error workflow are what stop a runaway LLM agent from exhausting API quotas before anyone notices; set them before the first real test run, not after an incident.
An LLM agent node breaks determinism, so design and test for the bounds, not the exact output — the same alert payload can trigger different tool-call sequences and different execution times on different runs; verify the workflow stays within its timeout and call limits, not that it produces identical output every time.
Measure what you assume, especially latency — a 30-minute POC run and an SSE connection-pool limit both turned out to be fixable (parallel MCP calls, migrating to Streamable HTTP), but only because I instrumented and looked at them directly instead of accepting "good enough for a proof of concept."

GitHub Repo

Full implementation: pipelineandprompts-labs/ai-in-the-stack/06-n8n-agentic-workflows

Includes the importable n8n workflow and a demo MCP server and RAG service, so you can run the whole thing locally without a live OpenShift cluster or GCP project. Also includes the error workflow and setup instructions for swapping in the real services.

What's Next

AI in the Stack #7 — Agentic AI Infrastructure: What It Takes to Do It Safely

This series has built every component of an AI-assisted operations stack — retrieval, live cluster access, governed prompts, portable providers, and now orchestrated automation. The final article covers what it actually takes, organizationally and architecturally, before any of this earns real operational authority: approval gates, blast-radius limits, and the governance model that decides when automation acts without a human in the loop.

Written by Pipeline & Prompts | Byte size guides on DevOps, Cloud and AI

Swapping LLM Providers Without Rewriting Your Stack

Nerav Doshi — Sun, 12 Jul 2026 22:48:45 +0000

🤖 AI in the Stack #5

Pipeline & Prompts | Byte size guides on DevOps, Cloud and AI

⚡ Byte Size Summary

Deploy a LiteLLM proxy on OpenShift to decouple your applications from individual LLM providers — switch or failover between models by changing a ConfigMap, not your code

Inject cluster-specific context (API versions, cloud annotations, golden templates) before every prompt to prevent models from generating "plausible averages" that look correct but fail on your platform

Validate every generated manifest with oc apply --dry-run=server before applying — the cluster's API server catches structural errors that no amount of prompt engineering will prevent

The Story

I was building a GenAI data poisoning demo for a proof of concept on ROSA (Red Hat OpenShift on AWS). The demo needed OpenShift-native manifests — Routes with TLS termination, proper SecurityContextConstraints, current API versions. I built it using Claude, and it worked. The model understood OpenShift. The manifests deployed cleanly. The demo ran on the cluster.

Then I ran out of free credits.

I was under a deadline, so I switched to OpenAI. The first manifest it generated used a Kubernetes Ingress instead of an OpenShift Route. The API versions referenced documentation that was two releases behind our cluster version. The YAML was syntactically valid. It was functionally wrong for OpenShift.

I tried to fix it by researching the correct documentation through NotebookLM, then feeding corrections back into the model. When that didn't converge, I switched to Gemini. Gemini gave me enough directional guidance that I could manually correct the manifests — using my own OpenShift knowledge, not the model's.

Two days later, I had a working demo again and ten-plus markdown files across my project. I couldn't tell which model had generated which file, which version was current, or which one had actually deployed successfully on the cluster. The project state had diverged so badly across three providers that rolling back to any previous working state was impossible — even if the credits had come back.

My model selection strategy during those two days was trial and error. I switched between Sonnet and Opus without understanding the difference between them. If a model's output deployed on the cluster, I stuck with it for that session. Every new session was a fresh gamble.

The ROSA cluster was my only validation oracle. If oc apply succeeded, the manifest stayed. If it errored, I had the model strip the broken section out and try again.

The failure wasn't switching providers. The failure was having no system in place that could make any provider produce correct output — no context injection, no validation gate, no abstraction layer. The fix is designing the control plane first — so models can be wrong without causing damage.

The Problem

LLMs generate statistically averaged infrastructure artifacts. They don't know your cluster — they know all clusters. The output is a blend of Kubernetes, OpenShift, ROSA, ARO (Azure Red Hat OpenShift), EKS (Amazon Elastic Kubernetes Service), and whatever else was in the training data. It looks right because it's an average of right things. An average of all correct configurations is correct for none of them.

This manifests in three ways:

Provider coupling. Your prompts are implicitly tuned to a specific model's training data and behavior. Switch providers — because of cost, credits, rate limits, or compliance — and the same prompt produces different output. Not always wrong output. Different output that may or may not be valid for your specific environment. You don't find out until you try to deploy it.

Silent knowledge gaps. A model that knows Kubernetes well may not know OpenShift. It generates a networking.k8s.io/v1 Ingress where route.openshift.io/v1 is what your cluster expects. It uses AWS load balancer annotations for a manifest targeting an ARO cluster. The output is valid YAML, valid Kubernetes — and wrong for your platform. There's no error message. The model doesn't know what it doesn't know.

No validation boundary. When a model generates a manifest, nothing in the default workflow checks whether it's correct for your cluster version, your cloud provider, or your installed operators. The human is the only validation layer — and under deadline pressure, the human trusts the output that looks right.

Why Existing Approaches Fall Short

"Just switch models." This was my first instinct. Claude ran out of credits, try OpenAI. OpenAI broke things, try Gemini. Each switch introduced different knowledge gaps without solving the underlying problem: no model has complete, current knowledge of your specific platform. Switching providers is moving the problem, not fixing it.

"Write better prompts." Prompt engineering helps with reasoning tasks — structuring output, defining roles, specifying constraints. It doesn't fix knowledge gaps. Telling a model "generate OpenShift-native manifests" doesn't inject the knowledge it needs about your cluster's API versions or your cloud provider's annotation requirements. If the information isn't in the model's training data, no prompt will extract it.

"Run it locally." I run Ollama on a 16GB Mac with models up to 14B parameters — Gemma 3 12B, DeepSeek-R1 14B, Qwen 2.5 Coder 14B. At that size, the models don't have meaningful OpenShift-specific knowledge. Loading a 9GB model on a machine with 16GB total — while running an IDE, terminal, and browser — means switching between models causes memory exhaustion. Local inference is a privacy-preserving fallback, not a primary working tier, for most practitioners on standard-issue hardware.

"Pick one provider and stay." Until you can't. Free credits expire mid-demo. Rate limits hit during a customer presentation. Your compliance team flags a data residency concern. A premium reasoning model handles many of the same tasks a cheap utility model could — paying for capability you don't need on every request adds up fast. The question isn't whether you'll need to switch providers — it's whether your system survives the switch when it happens.

The Architecture

The architecture has three layers, each preventing a different failure mode:

Layer 1 — Context Injection (Pre-processing). Before any prompt reaches a model, platform-specific context is injected: your cluster's API versions, your cloud provider's annotation requirements, your organization's golden templates. This separates "what the model knows" from "what your environment requires" — and fills the gap with verified context, not model guessing.

Layer 2 — LiteLLM Gateway (Routing). A LiteLLM proxy running in the ai-gateway namespace provides a single OpenAI-compatible API endpoint. Applications call the proxy, not individual providers. Model selection, failover, and tiering happen in the proxy configuration — not in application code. Three tiers:

Tier	Purpose	Characteristics
Utility	YAML generation, CRDs, JSON extraction, glue code	Cheap, fast, deterministic
Reasoning	Security analysis, architecture decisions, multi-hop logic	Slower, more expensive
Escape hatch	Edge cases, novel problems	Manual or explicit opt-in

Layer 3 — Post-Generation Validation (Post-processing). Every generated manifest runs through oc apply --dry-run=server before being applied. The cluster's API server validates structure, API versions, and field correctness against the actual cluster state — not against documentation that may be stale.

The key design insight: if a task is hard because of logic, route to a stronger model. If a task is hard because of domain knowledge, inject the missing context and keep it on a cheap model. Knowledge gaps are a context problem, not a model problem.

New to evaluating AI tooling on OpenShift? AI in the Stack #1 — AI Tooling on OpenShift: A Practitioner's Evaluation Framework covers the governance layer this gateway builds on.

How It Works: Step by Step

Prerequisites

OpenShift 4.14+ cluster (ROSA, ARO, or self-managed)
oc CLI 4.14+
At least one LLM provider API key (OpenAI, Anthropic, or both)
Namespace creation privileges on the cluster

Step 1 — Configure Model Tiers

The LiteLLM proxy configuration defines your model tiers and failover behavior:

# config.yaml — LiteLLM proxy configuration
model_list:
  - model_name: utility
    litellm_params:
      model: gpt-4.1-mini
      api_key: os.environ/OPENAI_API_KEY

  - model_name: utility-fallback
    litellm_params:
      model: claude-haiku-4-5-20251001
      api_key: os.environ/ANTHROPIC_API_KEY

  - model_name: reasoning
    litellm_params:
      model: claude-sonnet-5
      api_key: os.environ/ANTHROPIC_API_KEY

litellm_settings:
  fallbacks:
    - utility: [utility-fallback]
  num_retries: 2
  request_timeout: 30

router_settings:
  routing_strategy: simple-shuffle

The fallbacks setting means if the utility model fails — rate limit, timeout, provider outage — LiteLLM automatically retries with the fallback. Your application code doesn't change. It still sends requests to model: "utility" and the proxy handles the routing.

Step 2 — Build the Context Layer

Store platform-specific context in a ConfigMap. This is the knowledge your models lack — injected before every prompt, regardless of which provider handles the request:

# manifests/context-templates.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: openshift-context-templates
  namespace: ai-gateway
data:
  openshift-base.md: |
    You are generating manifests for an OpenShift 4.14+ cluster.

    Required API versions:
    - Deployments: apps/v1
    - Services: v1
    - Routes: route.openshift.io/v1
    - NetworkPolicies: networking.k8s.io/v1

    Rules:
    - Use OpenShift Routes, NOT Kubernetes Ingress
    - Never use deprecated APIs (extensions/v1beta1, apps/v1beta1)
    - Include resource requests and limits on all containers
    - Use securityContext with runAsNonRoot: true

  rosa-annotations.md: |
    Target platform: ROSA (Red Hat OpenShift on AWS)

    For Services of type LoadBalancer:
    - service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
    - service.beta.kubernetes.io/aws-load-balancer-scheme: "internal"

    Do NOT use Azure-specific annotations on this cluster.

  aro-annotations.md: |
    Target platform: ARO (Azure Red Hat OpenShift)

    For Services of type LoadBalancer:
    - service.beta.kubernetes.io/azure-load-balancer-internal: "true"

    Do NOT use AWS-specific annotations on this cluster.

Your application selects the appropriate context file based on the target cluster. On ROSA, prepend openshift-base.md and rosa-annotations.md to the system message. On ARO, swap in aro-annotations.md. The model never has to guess which cloud it's targeting.

Step 3 — Deploy the Gateway on OpenShift

oc new-project ai-gateway

Create the API key Secret — populate values from your external secret management system, never hardcode keys in manifests committed to Git:

# manifests/secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: llm-api-keys
  namespace: ai-gateway
type: Opaque
stringData:
  OPENAI_API_KEY: ""
  ANTHROPIC_API_KEY: ""

Deploy the proxy, service, and route:

# manifests/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: litellm-proxy
  namespace: ai-gateway
spec:
  replicas: 2
  selector:
    matchLabels:
      app: litellm-proxy
  template:
    metadata:
      labels:
        app: litellm-proxy
    spec:
      serviceAccountName: litellm-proxy
      containers:
        - name: litellm
          image: ghcr.io/berriai/litellm:main-latest
          ports:
            - containerPort: 4000
          envFrom:
            - secretRef:
                name: llm-api-keys
          volumeMounts:
            - name: config
              mountPath: /app/config.yaml
              subPath: config.yaml
              readOnly: true
          args: ["--config", "/app/config.yaml", "--port", "4000"]
          resources:
            requests:
              cpu: 250m
              memory: 512Mi
            limits:
              cpu: 500m
              memory: 1Gi
          livenessProbe:
            httpGet:
              path: /health
              port: 4000
            initialDelaySeconds: 15
            periodSeconds: 30
          readinessProbe:
            httpGet:
              path: /health
              port: 4000
            initialDelaySeconds: 5
            periodSeconds: 10
          securityContext:
            runAsNonRoot: true
            allowPrivilegeEscalation: false
            capabilities:
              drop: ["ALL"]
      volumes:
        - name: config
          configMap:
            name: litellm-config
---
apiVersion: v1
kind: Service
metadata:
  name: litellm-proxy
  namespace: ai-gateway
spec:
  selector:
    app: litellm-proxy
  ports:
    - port: 4000
      targetPort: 4000
---
apiVersion: route.openshift.io/v1
kind: Route
metadata:
  name: litellm-proxy
  namespace: ai-gateway
spec:
  to:
    kind: Service
    name: litellm-proxy
  port:
    targetPort: 4000
  tls:
    termination: edge

Load the configuration and deploy:

oc create configmap litellm-config \
  --from-file=config.yaml \
  -n ai-gateway

oc apply -f manifests/context-templates.yaml
oc apply -f manifests/secret.yaml
oc apply -f manifests/deployment.yaml

Verify the proxy is running:

oc get pods -n ai-gateway -l app=litellm-proxy
oc get route litellm-proxy -n ai-gateway -o jsonpath='{.spec.host}'

Step 4 — Wire Post-Generation Validation

The minimum viable safety net — run this after every manifest generation, before oc apply:

#!/bin/bash
# scripts/validate-manifest.sh
set -euo pipefail

MANIFEST="${1:?Usage: validate-manifest.sh <manifest.yaml>}"

echo "Validating against cluster API server..."
if oc apply --dry-run=server -f "$MANIFEST" 2>&1; then
  echo "PASS: Manifest accepted by $(oc whoami --show-server)"
else
  echo "FAIL: Manifest rejected — check API versions and resource types"
  exit 1
fi

What --dry-run=server catches:

API versions that don't exist on this cluster
Resource types not installed (missing CRDs — Custom Resource Definitions — or operators)
Required fields that are missing
Field type mismatches

What it does NOT catch:

Wrong cloud-provider annotations (annotations are unvalidated metadata)
Runtime behavior (a valid Deployment can still crash-loop)
Cross-resource dependency issues (Service referencing a non-existent Deployment)

For annotation validation on ROSA, add a targeted check:

# scripts/check-cloud-annotations.sh
if grep -q "azure-load-balancer" "$MANIFEST"; then
  echo "WARN: Azure LB annotation found — target cluster is ROSA"
  exit 1
fi

Step 5 — Lock Down Access

# manifests/rbac-and-network.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: litellm-proxy
  namespace: ai-gateway
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: litellm-proxy-access
  namespace: ai-gateway
spec:
  podSelector:
    matchLabels:
      app: litellm-proxy
  policyTypes:
    - Ingress
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              ai-gateway-access: "true"
      ports:
        - port: 4000
          protocol: TCP

Only namespaces with the ai-gateway-access: "true" label can reach the proxy. Label the namespaces that need access:

oc label namespace my-ai-workloads ai-gateway-access=true

Security and Operational Considerations

API keys belong in Secrets, not ConfigMaps. The llm-api-keys Secret is referenced via envFrom — keys are injected as environment variables, never mounted as files and never committed to Git. In production, populate this Secret from your external secret management system (Vault, AWS Secrets Manager, Azure Key Vault) using an operator like External Secrets.

Operational IDs leak infrastructure details. When troubleshooting with an LLM, you can scrub cluster names, VPC IDs, and IAM roles from the prompt context. You can't easily scrub operational IDs — error codes, correlation IDs, operation request IDs — because they're the thing you need the model to analyze. Those IDs can reveal internal infrastructure patterns, timing, and region information depending on the error format. For sensitive troubleshooting, route to local models and accept the latency cost. For everything else, the gateway's routing tier makes this a configuration choice, not a code change.

Infrastructure topology is the quieter data leak. Even without credentials, the manifests you send to external providers reveal your namespace naming conventions, RBAC (role-based access control) structures, network policy architecture, and CRD layouts. The context templates compound this — they describe your exact cluster configuration. Ensure context injection happens in-cluster (before the request leaves your network) so the platform knowledge stays on your infrastructure, even when the LLM call goes external.

The Route should use TLS termination appropriate to your environment. The example uses tls.termination: edge. For environments where the payload between the Route and the proxy must also be encrypted, use reencrypt and configure a serving certificate on the proxy container.

Restrict egress from the gateway namespace. The proxy needs outbound access to LLM provider APIs. It does not need access to your internal services. Add an egress NetworkPolicy that allows HTTPS to external endpoints and blocks everything else — this prevents the proxy from being used as an unintended hop into your internal network.

What Breaks at Scale

Cloud-specific annotations are the silent failure. In a multi-cluster deployment spanning ROSA and ARO, I've seen models confidently generate AWS Network Load Balancer annotations for manifests targeting an ARO cluster. The manifest passes --dry-run=server — annotations are unvalidated metadata. The failure surfaces only at runtime when traffic doesn't route correctly. The context templates per cluster are the mitigation, but they require discipline: every cluster needs its own context maintained, and the mapping from target cluster to correct context file must be automated, not manual.

Cluster versions diverge. Your staging cluster is on OpenShift 4.15. Production is still on 4.14 because the upgrade window hasn't opened. A manifest generated with 4.15 API assumptions may reference resources or fields that don't exist on 4.14. The --dry-run=server validation catches this — but only if you run it against the target cluster. Validating against staging and deploying to production defeats the purpose.

Knowledge fragments across teams. Five engineers, three different LLM providers, each with their own prompt patterns and context assumptions. The LiteLLM gateway standardizes the API layer. It doesn't standardize the context layer. Shared context templates in a ConfigMap help — but they require the same governance discipline as any other shared configuration artifact. Someone has to own the templates, review changes, and ensure they stay current when operators are upgraded or CRDs change.

LLMs smooth over differences instead of flagging them. This is the hardest failure mode to catch. A model generating manifests for a mixed ROSA/ARO environment doesn't flag the differences between the platforms — it generates something generic that works on neither. The output looks correct because it's a plausible blend of both platforms. The only defense is platform-specific context injection and a practitioner who knows enough to recognize when the model is giving a plausible average instead of a specific answer.

Quick Recap

Design the control plane before the first prompt — context injection, gateway routing, and post-generation validation catch errors that trial-and-error model switching cannot
oc apply --dry-run=server is your minimum viable safety net — it collapses a multi-day debugging spiral into a five-minute feedback loop, at zero infrastructure cost
Knowledge gaps are a context problem, not a model problem — inject verified platform context into cheap models instead of paying for expensive models that still guess

What's Next

AI in the Stack #6 — Building n8n Workflows for Platform Engineering Automation

This article kept the LLM on a short leash — every call went through a gateway, every output through a validation gate before it touched the cluster. The next one hands an LLM agent a longer leash inside an n8n workflow, wired to an MCP server and a RAG pipeline for automated incident triage. The tradeoff: the moment an agent node can decide how many tool calls to make, the workflow stops being deterministic. That article covers the execution limits you need before you trust it to run unattended.

Browse the rest of the AI in the Stack series index for the full run.

Written by Pipeline & Prompts | Byte size guides on DevOps, Cloud and AI