Nerav Doshi

Posted on Jul 22 • Originally published at pipelineandprompts.com

Build an AI Incident Triage Workflow in n8n with MCP

#n8n #openshift #incidentresponse #aiinthestack

🤖 AI in the Stack #6

Pipeline & Prompts | Byte size guides on DevOps, Cloud and AI

⚡ Byte Size Summary

Wire n8n to an MCP (Model Context Protocol) server over Streamable HTTP and a RAG (Retrieval-Augmented Generation) pipeline to build an automated incident triage workflow — runbook lookup and live cluster diagnostics assembled before a human opens a single dashboard

The moment you place an LLM agent node inside an n8n workflow, execution becomes non-deterministic — the model decides how many tool calls to make, so your timeout and call limits must assume worst case, not average case

Start with hard execution limits (5-minute timeout, 2-3 max calls per run) and loosen as you validate — a runaway workflow with no bounds will exhaust API quotas and flood notification channels before you notice

The Story

I started using n8n because I watched a teammate build a workflow in twenty minutes that would have taken me a day to write in Python. The visual canvas, the drag-and-drop nodes, the instant execution feedback — it made automation feel fast. So I started experimenting.

My first real workflows were personal — a financial portfolio tracker and a job search tool that scraped multiple sites. I pointed the job scraper at five websites simultaneously. All five came back with 401 errors — rate limited or blocked. The workflow had no error handling, no backoff, no maximum retry count. It kept firing. I only found out because I happened to check the n8n execution log in the UI and saw a wall of failed runs. No notification had told me anything was wrong.

Nothing critical broke. But five endpoints was enough to demonstrate the failure mode. You don't need fifty to create a problem when there's no bound on execution.

After several iterations on personal projects, I built something with real operational value: an MCP server for OpenShift SRE (Site Reliability Engineering) diagnostics. It exposes 9 read-only tools for cluster health, failing pods, events, storage, and CrashLoopBackOff diagnostics across ARO (Azure Red Hat OpenShift), ROSA HCP (Red Hat OpenShift Service on AWS, Hosted Control Plane), and OSD-GCP (OpenShift Dedicated on Google Cloud). Paired it with a RAG knowledge base loaded with runbooks and SOPs. Both worked as standalone tools.

The question was obvious: what happens when an alert fires and both tools run automatically, in coordination, before I've opened a single dashboard? That's what n8n was for — the orchestration layer that connects the pieces this series has built.

What I didn't expect was how fundamentally the workflow would change once I put an LLM agent inside it.

The Problem

Every component built in this series so far — RAG runbook retrieval (Article 02), MCP cluster diagnostics (Article 03), versioned prompts (Article 04), provider abstraction (Article 05) — is something a human calls manually. An engineer gets an alert, opens a terminal, queries the RAG pipeline, checks the cluster, cross-references the runbook, and posts a summary to Slack. That process takes 15-30 minutes per incident, and it requires the engineer to know which tools to call in which order.

The orchestration gap isn't a tooling problem. Every individual tool works. The gap is that nothing connects them into an automated response that fires before the engineer has context-switched from whatever they were doing.

But there's a deeper problem. Traditional workflow automation — a webhook triggers a script, the script runs steps in order, the output is predictable — is deterministic. Step A always leads to Step B, the number of API calls is fixed, the execution time is bounded by design.

Add an LLM agent to that workflow and determinism disappears. The agent decides how many MCP tools to call based on its interpretation of the input. It might call get_cluster_health once, or it might call get_failing_pods, then get_events, then diagnose_crashloop — three calls instead of one, against live cluster APIs, with execution time that varies per run. Your workflow looks the same on the canvas. Its runtime behavior is different every time.

Why Existing Approaches Fall Short

Custom Python scripts. You can wire RAG + MCP + Slack in an async Python service. I've done it. Managing concurrent HTTP calls, retry logic, error handling, and state across a multi-step async chain produces code that's hard to debug and hard to hand off to a teammate. It's also invisible when it fails silently at 3am. n8n's visual execution log shows exactly which node failed and what data it received — that visibility alone justified the switch.

GitHub Actions. Good for CI/CD, wrong for real-time event response. Actions are stateless, trigger on repository events, and have cold-start latency. An incident triage workflow needs to fire on a Prometheus webhook, hold state between steps, and complete in seconds to minutes — not wait for a runner to spin up.

Zapier or cloud-hosted automation. Not self-hostable. Your workflow data, your cluster diagnostics, your runbook content — all leave your infrastructure boundary. For platform engineering on OpenShift, that's a non-starter in most enterprise environments.

n8n without guardrails. This is the approach I started with, and it's the one that produced the runaway loop. n8n makes building workflows remarkably easy. That ease is precisely why it needs execution limits, error workflows, and bounded retries before the workflow goes anywhere near a production alerting pipeline.

The Architecture

The architecture connects four components through n8n as the orchestration layer:

Trigger. A webhook endpoint in n8n receives an alert payload. For the POC, this was manually triggered with curl. The production target is a Prometheus Alertmanager webhook — configuration provided below, but not yet tested against a live Alertmanager.

RAG Service. The runbook retrieval pipeline from Article 02. n8n calls it via HTTP Request node, passing the alert description as the query. Returns relevant runbook content and source references.

MCP Server. The OpenShift SRE diagnostic server, connected via n8n's community MCP client node over Streamable HTTP transport. The MCP spec deprecated the older SSE (Server-Sent Events) transport in its 2025-03-26 revision, so this article builds on Streamable HTTP throughout — see the transport note in Step 1 if your server hasn't migrated yet. It provides live cluster state — failing pods, recent events, cluster health — without custom REST wrappers or protocol translation.

LLM (Vertex AI). AI analysis runs through Google Vertex AI, configured using n8n's built-in GCP credentials. I started with Ollama locally but the 16GB hardware constraint made it unusable for real workflows — same limitation from Article 05. Vertex AI through GCP was the path that worked.

Slack. The output destination. A single message combining runbook guidance and live cluster state, posted before a human has opened a dashboard.

The three lookups — runbook retrieval, failing-pod check, and recent-events check — run concurrently instead of one waiting on another. That's a deliberate change from an earlier version of this workflow that routed both MCP calls through a single combined node. Two independent MCP Client nodes, one per tool, can execute in parallel — one node handling both tools couldn't.

How It Works: Step by Step

Prerequisites

n8n self-hosted instance (local or on-cluster)
The community node package n8n-nodes-mcp installed in that instance (Settings → Community Nodes → Install → n8n-nodes-mcp) — not n8n's built-in MCP Client Tool node, which only works as an AI Agent sub-node and can't be wired the way this workflow needs. This is a self-hosted-only step; it's not available on n8n Cloud.
MCP server running with Streamable HTTP transport (openshift-mcp-sre-tools — confirm which transport your deployment currently runs; the MCP spec deprecates SSE, so verify before assuming Streamable HTTP support)
RAG service running (Article 02 pipeline)
Google Cloud project with Vertex AI API enabled
GCP service account credentials configured in n8n
Slack workspace with an incoming webhook or bot token

Step 1 — Connect n8n to the MCP Server

n8n's built-in MCP Client Tool node connects to an MCP server, but it's a sub-node that plugs into an AI Agent's tool input. It isn't a standalone node with regular output connections you can wire directly into a Merge step. This workflow calls get_failing_pods and get_events deterministically, outside of agent reasoning, so it uses the community n8n-nodes-mcp package's standalone MCP Client node instead, which supports direct calls with normal input/output connections. Install that package (see Prerequisites) before following the steps below — the built-in node won't let you wire the canvas this way.

A separate transport note, since it affects which credential fields you fill in: the MCP spec deprecated the HTTP+SSE transport in its 2025-03-26 revision in favor of Streamable HTTP. Client-side SSE support has been narrowing since. Use Streamable HTTP for any new MCP server; only fall back to SSE if you're connecting to an existing server that hasn't migrated yet.

Unlike an HTTP Request node, the server endpoint isn't a field on the MCP Client node itself — it lives in a credential, shared across every MCP Client node that uses it:

In n8n's Credentials panel, create a new credential of type MCP Client API (this type appears once the community package is installed). It offers a connection-type selector — choose HTTP Streamable.
Set the HTTP Streamable URL to your MCP server's address, e.g. http://platform-mcp-server:8080/mcp.
Save it with a name you'll recognize — this article uses Platform MCP Server (Streamable HTTP).

Then, for each MCP tool you want to call, add an MCP Client node (from n8n-nodes-mcp) to the canvas:

Resource: Tool
Operation: Execute Tool
Tool Name: the specific tool to call, e.g. get_failing_pods
Credential: the Platform MCP Server (Streamable HTTP) credential from above

This workflow needs two calls — get_failing_pods and get_events — so add two separate MCP Client nodes, both pointed at the same credential, each with its own Tool Name. That's also what makes the parallel-execution pattern in Step 3 possible: two independent nodes can run concurrently, where one combined "call whatever tools you need" node couldn't.

The MCP server runs as a separate process — either on your local machine during development or as a pod on OpenShift for production. n8n connects to it as a network service over HTTP, not as a spawned child process. This means the MCP server has its own lifecycle, scaling, and RBAC independent of n8n.

Confirmed: this workflow uses the community n8n-nodes-mcp package's standalone MCP Client node, not the built-in n8n-nodes-langchain.mcpClientTool. That's the only one of the two that supports the direct, deterministic tool calls this workflow relies on — the built-in node is agent-only and can't be wired into a Merge step this way.

Step 2 — Configure Vertex AI Credentials

In n8n's Credentials panel, add a Google Cloud credential:

Upload or paste your GCP service account JSON key
The credential provides access to Vertex AI endpoints in your project

Add a Google Vertex Chat Model node to the canvas (n8n's docs also refer to it as "Google Vertex AI Chat Model" — same node):

Project ID: your GCP project
Region: us-central1 (or your preferred region) — note that as of this writing n8n's Vertex node doesn't expose a region/zone field directly in all versions; if yours doesn't, region falls back to your GCP project default, which can cause quota errors in unexpected regions. Check your installed version's node parameters before assuming this field exists.
Model: the model available in your Vertex AI project

This replaces any direct Anthropic or OpenAI API call. All LLM requests route through your GCP project, using your organization's billing, IAM policies, and data residency controls.

Step 3 — Build the Workflow

The Trigger. Add a Webhook node as the entry point. For testing, trigger it manually with curl:

curl -X POST http://localhost:5678/webhook/incident-triage \
  -H "Content-Type: application/json" \
  -d '{
    "alerts": [{
      "status": "firing",
      "labels": {
        "alertname": "PodCrashLooping",
        "severity": "warning",
        "namespace": "production",
        "service": "payment-api"
      },
      "annotations": {
        "description": "Pod payment-api-7d4b8c6f5-x2k9m has restarted 5 times in 10 minutes"
      },
      "startsAt": "2026-07-20T14:32:00Z"
    }]
  }'

For production, configure Alertmanager to call this webhook — but note this was not tested against a live Alertmanager in the POC. This snippet shows the receiver only; it still needs a matching route: block that selects this receiver by label match, or Alertmanager won't send anything to it:

# alertmanager.yml — production target (not yet tested)
receivers:
  - name: n8n-platform-automation
    webhook_configs:
      - url: 'https://your-n8n-instance/webhook/incident-triage'
        send_resolved: true

# route: block required elsewhere in the config to actually invoke this receiver, e.g.:
# route:
#   routes:
#     - receiver: n8n-platform-automation
#       matchers:
#         - severity =~ "warning|critical"

Parse the alert. Add a Code node to extract fields from the Alertmanager payload format:

const alert = $input.first().json.alerts[0];

return {
  json: {
    alertname: alert.labels.alertname,
    severity: alert.labels.severity,
    namespace: alert.labels.namespace,
    service: alert.labels.service || alert.labels.deployment || "unknown",
    status: alert.status,
    description: alert.annotations.description || alert.annotations.summary,
    startsAt: alert.startsAt
  }
};

Query the RAG pipeline. Add an HTTP Request node calling the RAG service from Article 02, and name it RAG Query on the canvas to match the code references in later steps:

Method: POST
URL: http://runbook-rag-service:8080/query
Body (JSON):
{
  "question": "How do I troubleshoot {{ $json.alertname }} for {{ $json.service }}? {{ $json.description }}"
}

Query the MCP server. Add the two MCP Client nodes configured in Step 1, both fed directly from Parse Alert so they run in parallel rather than one waiting on the other:

Name one MCP Get Failing Pods, with Tool Name set to get_failing_pods
Name the other MCP Get Events, with Tool Name set to get_events

Name them exactly this way — the code in the next steps references both nodes by these exact names, and n8n's $('NodeName') expression syntax requires an exact match. Both pass the namespace from the parsed alert as the tool parameter.

The bounded conditional. Add an IF node to check whether the issue has already self-resolved before escalating:

Condition: MCP response shows no failing pods in the namespace

True branch: lightweight "auto-resolved" notification. False branch: full alert with runbook guidance and cluster state.

This single conditional separates "automation that helps" from "automation that floods a channel with noise every time a transient blip self-corrects."

Format and post to Slack. Add a Code node to merge the RAG, failing-pods, and events responses:

const ragAnswer = $('RAG Query').first().json.answer;
const ragSources = $('RAG Query').first().json.sources.join(', ');
const failingPods = $('MCP Get Failing Pods').first().json.failing_pods;
const recentEvents = $('MCP Get Events').first().json.recent_events;
const alert = $('Parse Alert').first().json;

const failingPodsText = failingPods.length > 0
  ? failingPods.map(p => `• ${p.name} (${p.phase})`).join('\n')
  : 'None currently failing';

return {
  json: {
    text: `*${alert.alertname}* — ${alert.service} (${alert.namespace})\n\n` +
          `*Runbook guidance:*\n${ragAnswer}\n_Source: ${ragSources}_\n\n` +
          `*Current cluster state:*\n${failingPodsText}\n\n` +
          `*Recent events:* ${recentEvents.length} events in last check`
  }
};

[TECHNICAL FIX: the original code referenced $json.alertname here, which only resolves against the item arriving at this specific node. After the Merge and Analyze steps, that item is whatever those nodes output, not the original parsed alert. Referencing $('Parse Alert').first().json explicitly is the correct n8n pattern for pulling data from an earlier node in the chain, regardless of what's passed through the nodes in between. It also originally referenced a single $('MCP Query') node, from before the MCP call was split into two parallel nodes — updated to pull from MCP Get Failing Pods and MCP Get Events directly.]

This is the payoff. The Slack message an on-call engineer receives contains grounded runbook guidance and live cluster state, assembled automatically, before they have opened a single tool.

Step 4 — Set Execution Limits

Three settings prevent the runaway loop failure mode:

Execution timeout. In n8n's Workflow Settings, set a maximum execution time. Start with 5 minutes for experimentation. The POC workflow took approximately 30 minutes end-to-end — mostly due to Kubernetes API query latency across multiple clusters via the MCP server. That's acceptable for a proof of concept. It is not acceptable for production incident response. Design around this latency before moving to production: limit the number of clusters queried per run, cache cluster state, or run MCP queries in parallel.

Retry limits on HTTP nodes. Use n8n's built-in retry configuration on HTTP Request nodes — not manual loops:

HTTP Request node settings:
  Retry on Fail: enabled
  Max Tries: 3
  Wait Between Tries: 2000ms

The built-in retry mechanism cannot become infinite. A manual loop with a Wait node and a conditional back-edge can — as I learned on the scraping workflow.

Error workflow. In Workflow Settings, set an Error Workflow — a separate, simple workflow that fires if the main workflow throws an exception. At minimum, have it post to a dedicated #automation-errors Slack channel. A failure in your incident automation should never fail silently.

Step 5 — Add Execution Logging

Add a final Code node that logs a structured summary of every run:

return {
  json: {
    timestamp: new Date().toISOString(),
    alertname: $('Parse Alert').first().json.alertname,
    namespace: $('Parse Alert').first().json.namespace,
    auto_resolved: $('MCP Get Failing Pods').first().json.failing_pods.length === 0,
    workflow_execution_id: $execution.id
  }
};

[TECHNICAL FIX: the original code referenced $json.alertname and $json.is_healthy. is_healthy was never set by any earlier node in this workflow — it would have thrown a runtime error or silently returned undefined depending on n8n's expression handling. Both fields now pull from nodes that actually produced them.]

This gives you the audit trail this series has insisted on since Article 03: every automated action should be traceable after the fact, not just visible while it's running.

Security and Operational Considerations

The MCP server must be read-only. The openshift-mcp-sre-tools server runs in read-only mode by default — all 9 tools are diagnostic. This is the design property that makes it safe to wire into n8n without additional guardrails. If those tools could modify cluster state, an LLM agent deciding to call them unpredictably would be a fundamentally different risk profile. Start read-only. Add write capabilities only when you have execution limits, approval gates, and rollback mechanisms in place.

GCP credentials in n8n need protection. n8n stores credentials encrypted at rest, but the n8n instance itself becomes a high-value target. It holds GCP service account keys that access Vertex AI and potentially other GCP services. If n8n runs on-cluster, the namespace needs a NetworkPolicy restricting who can reach the n8n UI and API. If it runs locally, access is inherently restricted to the operator's machine. Scope the service account to a minimal custom IAM (Identity and Access Management) role — Vertex AI User at most, not Editor or Owner. A broadly-scoped key stolen from n8n's credential store is a much bigger blast radius than a Vertex-AI-only one.

The webhook endpoint is an attack surface. When n8n runs on-cluster with a webhook trigger, that endpoint needs to be reachable by Alertmanager but not by the public internet. Use OpenShift Routes with edge TLS termination (spec.tls.termination: edge) and restrict access via NetworkPolicy. Authenticate the webhook — Alertmanager supports basic auth on webhook configs — so that arbitrary POST requests can't trigger your automation. Authentication alone doesn't bound volume, though. Also rate-limit the webhook, at the Route or via an n8n-side check. That way a real alert storm from a misconfigured Alertmanager rule can't multiply into dozens of concurrent LLM-agent executions and exhaust the same API quotas the execution limits in Step 4 are meant to protect.

MCP server connection security. The Streamable HTTP transport between n8n and the MCP server should run over TLS if they're on separate hosts. If both run in the same cluster, a ClusterIP Service with a NetworkPolicy restricting ingress to the n8n namespace is sufficient. If you're still running an MCP server on the deprecated SSE transport, the same TLS/NetworkPolicy guidance applies — but plan the migration to Streamable HTTP regardless, since SSE client support will keep narrowing.

What the workflow sends externally. The RAG query and the MCP cluster diagnostics — namespace names, pod names, event messages — are sent to Vertex AI for analysis. These aren't credentials, but they reveal internal infrastructure details. The same data leakage consideration from Article 05 applies: Vertex AI runs in your GCP project, which is better than a third-party provider API, but the data still leaves the OpenShift cluster boundary.

Audit trail durability. The execution logging in Step 5 captures a structured summary per run, but that's only useful as an audit trail if it's actually retained. n8n prunes execution data on a configurable schedule, and the default retention window is short. This automation should be auditable after the fact — it's an incident-response workflow touching production clusters. Explicitly configure execution data saving and retention in n8n's settings, or ship the Log Execution node's output to an external log store (e.g., OpenShift's log aggregation stack) instead of relying on n8n's own execution history.

Multi-tenant Slack exposure. This workflow posts namespace names, pod names, and event messages to Slack. If the destination channel is shared across teams, or the alert can originate from any namespace on a multi-tenant cluster, that's cross-tenant data exposure through the notification — even though the MCP server itself is read-only and correctly scoped. Route notifications to a channel scoped to the alert's originating namespace or team, not one shared channel for every cluster the MCP server can see.

What Breaks at Scale

30-minute latency is not production-ready. The POC workflow took approximately 30 minutes end-to-end. The bottleneck was Kubernetes API queries across multiple clusters via the MCP server. For a proof of concept that validates the flow, this is acceptable. For an incident response workflow that needs to beat a human to the diagnosis, it's too slow. The production path requires limiting clusters per query, parallelizing MCP calls, or caching recent cluster state.

Non-deterministic execution makes testing harder. A traditional n8n workflow produces the same output for the same input every time. With an LLM agent node, the same alert payload can trigger different MCP tool call sequences, different analysis depth, and different token consumption per run. You can't write deterministic integration tests against this workflow. You test the bounds — does it stay within the timeout, does it stay within the call limit — not the exact output.

Transport choice affects concurrent-user scaling, but isn't the only bottleneck. An earlier version of this workflow used the SSE transport, which holds one persistent connection open per workflow execution. Five engineers running concurrent workflows meant five persistent connections, and the MCP server's connection pool filled up under load. That specific problem is largely why the MCP spec deprecated SSE in favor of Streamable HTTP, which supports a stateless request/response mode designed for concurrent, load-balanced access — this article now builds on Streamable HTTP for that reason. What Streamable HTTP doesn't fix: every MCP call still triggers real Kubernetes API queries against real clusters, and cloud provider API throttling (Azure for ARO, AWS for ROSA) sits upstream of the transport layer entirely. Fixing the connection model doesn't fix cluster API rate limits. This architecture still works best for a single operator or a small team; at real scale, the Kubernetes API layer — not the MCP transport — is the first bottleneck you'll hit.

Visual simplicity hides execution complexity. The n8n canvas shows five or six nodes in a clean flow. What it doesn't show is the LLM agent making variable numbers of tool calls per run, each with its own latency, retry behavior, and failure mode. The canvas is the map, not the territory. The execution log is where you see what actually happened.

What I'd Do Differently

Two things, both cheap to implement on day one.

Hard execution limits before the first test run. Five-minute timeout, 2-3 maximum tool calls per workflow execution. Not because those are the right numbers for production — I don't know the right numbers yet, and finding them is trial and error. But having any limit at all would have caught the runaway scraper in minutes instead of hours. Start tight, hit the limits during real runs, adjust based on what you observe. The important thing is that limits exist, not that they're perfect.

Design around latency from the start. I accepted 30-minute workflow runs because I was validating the flow, not optimizing it. The honest lesson is that I should have instrumented execution time per node from the first run and identified the MCP cluster query bottleneck immediately. Then I could have made a conscious decision about whether to accept it or fix it, instead of discovering it casually after multiple runs.

Quick Recap

Guardrails come before the happy path — an execution timeout, retry limits, and a dedicated error workflow are what stop a runaway LLM agent from exhausting API quotas before anyone notices; set them before the first real test run, not after an incident.
An LLM agent node breaks determinism, so design and test for the bounds, not the exact output — the same alert payload can trigger different tool-call sequences and different execution times on different runs; verify the workflow stays within its timeout and call limits, not that it produces identical output every time.
Measure what you assume, especially latency — a 30-minute POC run and an SSE connection-pool limit both turned out to be fixable (parallel MCP calls, migrating to Streamable HTTP), but only because I instrumented and looked at them directly instead of accepting "good enough for a proof of concept."

GitHub Repo

Full implementation: pipelineandprompts-labs/ai-in-the-stack/06-n8n-agentic-workflows

Includes the importable n8n workflow and a demo MCP server and RAG service, so you can run the whole thing locally without a live OpenShift cluster or GCP project. Also includes the error workflow and setup instructions for swapping in the real services.

What's Next

AI in the Stack #7 — Agentic AI Infrastructure: What It Takes to Do It Safely

This series has built every component of an AI-assisted operations stack — retrieval, live cluster access, governed prompts, portable providers, and now orchestrated automation. The final article covers what it actually takes, organizationally and architecturally, before any of this earns real operational authority: approval gates, blast-radius limits, and the governance model that decides when automation acts without a human in the loop.

Written by Pipeline & Prompts | Byte size guides on DevOps, Cloud and AI

DEV Community