DEV Community

Mateo Ruiz
Mateo Ruiz

Posted on Originally published at itpathsolutions.com

Build an AI Agent Error Triage Pipeline with n8n, Gemini & Slack

#n8n #ai #automation #devops

When LLM agents hit production traffic, manual error triage stops scaling fast. The same five failure types keep appearing under different traces, each re-diagnosed from scratch. Duplicate alerts flood Slack. Engineers start muting the channel.

This post walks through a production-ready n8n workflow that reads a raw agent error, deduplicates it, classifies it against five real failure categories using Gemini, attaches a fix playbook, and posts a severity-scored triage report to Slack — without forcing a classification when the model isn't confident.

Free workflow JSON at the end.

Why Manual AI Agent Triage Breaks at Scale

LLM agent failures cluster into a small number of repeatable categories:

Category Description
Tool Call Failure External API, database, or function call failed outright
Context Window Exhaustion Conversation or tool history exceeded model context limit
State Corruption Agent memory/task state in unanticipated condition
Retry Loop Agent stuck repeating the same failed step without progress
Uncertain Error doesn't cleanly fit any named category

The problem: nothing remembers having seen these before. Same failure type, fresh diagnosis, every time. One bug generates dozens of identical alerts. Without severity signals, everything looks equally urgent — which means nothing is — and engineers tune out.

Architecture 

┌─────────────────────────────────────────────────────────────┐
│              n8n AI Agent Triage Workflow                    │
│                                                             │
│  [Webhook: POST /agent-error-triage]                        │
│       ↓                                                     │
│  Accepts: Sentry native payload OR { message: "..." }       │
│       ↓                                                     │
│  [Extract & Dedup Code Node]                                │
│       → Normalize payload (title, culprit, stack_trace)     │
│       → Build fingerprint (hash of identifying fields)      │
│       → Check 30-min in-memory suppression window           │
│       ↓              ↓                                      │
│  [isDuplicate=true]  [isDuplicate=false]                    │
│       ↓                    ↓                                │
│  [Skip No-Op]    [Classify with Gemini]                     │
│                       ↓                                     │
│              Structured Output Parser                        │
│              → { category, confidence, severity,            │
│                  summary, reasoning }                        │
│                       ↓                                     │
│              [Build Triage Report Code Node]                │
│              → Map category → fix playbook                  │
│              → Assemble Slack message                       │
│                       ↓                                     │
│              [Post Triage Report: Slack Node]               │
│              → #agent-alerts channel                        │
└─────────────────────────────────────────────────────────────┘
Enter fullscreen mode Exit fullscreen mode

Stack

Tool Role
n8n (self-hosted) Orchestration
Sentry (optional) Error source via webhook
Google Gemini AI classification
LangChain in n8n Chain + Structured Output Parser
Slack Bot Triage report delivery

Node-by-Node Breakdown

1. Webhook Trigger

Listens on POST /agent-error-triage. Accepts two payload shapes:

Sentry native webhook:

{
  "data": {
    "event": {
      "title": "ToolCallError: Timeout on /api/crm",
      "culprit": "agent.tool_executor.call_external",
      "exception": {
        "values": [{ "stacktrace": { "frames": [...] } }]
      }
    }
  }
}
Enter fullscreen mode Exit fullscreen mode

Generic payload (any error source):

{
  "message": "Agent failed: context limit exceeded after 47 tool calls"
}
Enter fullscreen mode Exit fullscreen mode

Both shapes are normalised to the same internal structure in the next node.

2. Extract & Dedup Code Node

This is the most important node in the pipeline. It does two things:

Normalise the payload:

// Code Node — Extract & Dedup
const input = $input.first().json;

// Handle both Sentry and generic payloads
let title, culprit, stackTrace;

if (input.data?.event) {
  // Sentry native format
  const event = input.data.event;
  title = event.title || 'Unknown Error';
  culprit = event.culprit || '';
  const frames = event.exception?.values?.[0]?.stacktrace?.frames || [];
  stackTrace = frames.map(f => `${f.filename}:${f.lineno} in ${f.function}`).join('\n');
} else {
  // Generic format
  title = input.message || 'Unknown Error';
  culprit = input.culprit || '';
  stackTrace = input.stack_trace || '';
}

// Build fingerprint — hash of identifying fields (not timestamps/IDs)
const fingerprintSource = `${title}::${culprit}::${stackTrace.substring(0, 200)}`;
const fingerprint = fingerprintSource.split('').reduce((hash, char) => {
  return ((hash << 5) - hash) + char.charCodeAt(0);
}, 0).toString(36);

// Check 30-minute suppression window using n8n static data
const staticData = $getWorkflowStaticData('global');
const seen = staticData.seenFingerprints || {};
const now = Date.now();
const windowMs = 30 * 60 * 1000; // 30 minutes

// Clean expired entries
Object.keys(seen).forEach(fp => {
  if (now - seen[fp] > windowMs) delete seen[fp];
});

const isDuplicate = !!seen[fingerprint];

if (!isDuplicate) {
  seen[fingerprint] = now;
}

staticData.seenFingerprints = seen;

return [{
  json: { title, culprit, stackTrace, fingerprint, isDuplicate }
}];
Enter fullscreen mode Exit fullscreen mode

Why non-cryptographic hashing: MD5/SHA isn't available in n8n's sandboxed JS environment by default. The djb2-style hash above is sufficient for fingerprinting — collision probability across a 30-minute window is negligible for error triage purposes.

3. IF Node — Route Duplicates 

isDuplicate == true  → Skip (No-Op node, logs fingerprint)
isDuplicate == false → Continue to classification
Enter fullscreen mode Exit fullscreen mode

This is what prevents one bug from generating forty identical Slack reports.

4. Classify with Gemini (LangChain Chain Node)

System Prompt 

You are an AI agent error classifier. Classify the error into exactly one of these categories:

1. tool_call_failure — A call to an external API, database, function, or tool failed outright. Look for timeout errors, HTTP 4xx/5xx codes, connection refused, or authentication failures in tool calls.

2. context_window_exhaustion — The model's context limit was reached. Look for "context length exceeded", "max tokens", or errors after long chains of tool calls with no summarisation.

3. state_corruption — The agent's internal memory or task state is in an unexpected condition. Look for key errors on expected state fields, type mismatches in state objects, or missing required state values.

4. retry_loop — The agent is stuck repeating the same failed action without progress. Look for repeated identical tool calls, loop detection errors, or max retry exceeded messages.

5. uncertain — The error doesn't clearly fit any category above. Use this when evidence is ambiguous or the error message lacks enough context to classify confidently.

Rules:
- Choose uncertain rather than force a bad fit
- Base classification only on evidence in the error, not assumptions
- Confidence should reflect actual evidence strength, not optimism

Return ONLY valid JSON:
{
  "category": "tool_call_failure|context_window_exhaustion|state_corruption|retry_loop|uncertain",
  "confidence": 0.0-1.0,
  "severity": "low|medium|high|critical",
  "summary": "One sentence plain-language description of what happened",
  "reasoning": "Why you chose this category based on specific evidence in the error"
}
Enter fullscreen mode Exit fullscreen mode

Key settings:

  • Temperature: 0.1 — keeps labels consistent across identical errors regardless of time-of-day
  • Model: gemini-1.5-flash or gemini-1.5-pro — confirm availability against your live credential before deploying
  • Max output tokens: 512 — sufficient for the JSON response

Structured Output Parser

The LangChain Structured Output Parser sub-node enforces the JSON schema. If parsing fails (model added markdown fences, malformed JSON, etc.), route to the error branch rather than letting bad output propagate downstream.

5. Build Triage Report Code Node

Maps category to fix playbook and assembles the Slack message:

// Code Node — Build Triage Report
const { category, confidence, severity, summary, reasoning } = $input.first().json;
const { title, culprit, stackTrace } = $('Extract & Dedup').first().json;

const playbooks = {
  tool_call_failure: [
    '1. Check external service status page for outages',
    '2. Review tool call logs for HTTP status codes and timeouts',
    '3. Verify API credentials and rate limit headers',
    '4. Add retry with exponential backoff if not already present'
  ],
  context_window_exhaustion: [
    '1. Add conversation summarisation step before context limit is reached',
    '2. Review tool call chain length — reduce unnecessary calls',
    '3. Implement sliding window for conversation history',
    '4. Consider chunking long tool outputs before adding to context'
  ],
  state_corruption: [
    '1. Add state validation checks at each agent step',
    '2. Review state schema for missing required fields',
    '3. Check for race conditions in concurrent agent runs',
    '4. Add state recovery logic or restart from last valid checkpoint'
  ],
  retry_loop: [
    '1. Implement loop detection — track repeated tool call signatures',
    '2. Set max retry limit with circuit breaker',
    '3. Add fallback action when retry limit is reached',
    '4. Review condition for loop exit — is the success criteria reachable?'
  ],
  uncertain: [
    '1. Review full stack trace manually — insufficient context for classification',
    '2. Improve agent error logging to include more diagnostic detail',
    '3. Check recent deployments for changes that could explain the failure',
    '4. Escalate to senior engineer if pattern recurs'
  ]
};

const severityEmoji = {
  critical: '🔴',
  high: '🟠',
  medium: '🟡',
  low: '🟢'
};

const confidenceBar = confidence >= 0.8 ? 'High' : confidence >= 0.5 ? 'Medium' : 'Low';
const playbook = playbooks[category] || playbooks.uncertain;

// Build Slack Block Kit message
const blocks = [
  {
    type: 'header',
    text: {
      type: 'plain_text',
      text: `${severityEmoji[severity] || '⚪'} AI Agent Error Triage Report`
    }
  },
  {
    type: 'section',
    fields: [
      { type: 'mrkdwn', text: `*Category:*\n${category.replace(/_/g, ' ').toUpperCase()}` },
      { type: 'mrkdwn', text: `*Severity:*\n${severity.toUpperCase()}` },
      { type: 'mrkdwn', text: `*Confidence:*\n${confidenceBar} (${(confidence * 100).toFixed(0)}%)` },
      { type: 'mrkdwn', text: `*Culprit:*\n${culprit || 'Unknown'}` }
    ]
  },
  {
    type: 'section',
    text: { type: 'mrkdwn', text: `*Error:*\n${title}` }
  },
  {
    type: 'section',
    text: { type: 'mrkdwn', text: `*Summary:*\n${summary}` }
  },
  {
    type: 'section',
    text: { type: 'mrkdwn', text: `*AI Reasoning:*\n${reasoning}` }
  },
  {
    type: 'section',
    text: {
      type: 'mrkdwn',
      text: `*Suggested Fix Steps:*\n${playbook.join('\n')}`
    }
  },
  { type: 'divider' }
];

return [{ json: { blocks, category, severity, confidence } }];
Enter fullscreen mode Exit fullscreen mode

6. Post Triage Report — Slack Node

Configure the Slack node with:

  • Authentication: Bot token (scoped to one channel only — not workspace-wide)
  • Channel: #agent-alerts channel ID
  • Message type: Blocks (pass the blocks array from previous node)

Slack bot scoping — important: The bot token should be limited to chat:write in the specific channel. Do not use a token with broad workspace permissions for this integration.

7. Error Handling

Low confidence or JSON parse failures should not silently drop. Build an error branch:

// Error branch — uncertain fallback
const errorContext = {
  timestamp: new Date().toISOString(),
  stage: $input.first().json.failedNode || 'classification',
  rawError: $input.first().json.error?.message || 'Unknown',
  originalTitle: $('Extract & Dedup').first().json?.title || 'Unknown'
};

// Override with uncertain classification
return [{
  json: {
    category: 'uncertain',
    confidence: 0,
    severity: 'medium',
    summary: 'Classification failed — manual review required',
    reasoning: `Workflow error at ${errorContext.stage}: ${errorContext.rawError}`,
    ...errorContext
  }
}];
Enter fullscreen mode Exit fullscreen mode

Route this into the Build Triage Report node so the uncertain playbook still goes to Slack — a failed classification is still an incident that needs visibility.

Customisation Options

Swap the AI Provider

Gemini isn't a hard requirement. Replacing it means updating the credential and model node — the Structured Output Parser is model-agnostic as long as the JSON schema is valid. Claude and GPT-4o both work well for structured classification tasks.

Note on Gemini rate limits: Free tier limits vary by model version and can change. For production traffic, confirm current limits before relying on the free tier — hitting a rate limit mid-triage means classification requests silently fail.

Adjust the Dedup Window

The 30-minute window is a starting point. If the same bug tends to resurface after fixes and rollbacks, a shorter window makes sense. If your agents produce very infrequent errors, a longer window reduces noise further.

const windowMs = 30 * 60 * 1000; // Adjust this value
Enter fullscreen mode Exit fullscreen mode

Add an Approval Gate (Regulated Environments)

Currently everything posts to Slack automatically. To add a human gate before high-severity classifications trigger downstream automation:

[Build Triage Report]
       ↓
[IF severity == critical]
       ↓
[Send approval email: Proceed / Escalate]
       ↓
[n8n Wait Node]
       ↓
[Webhook: receives decision]
       ↓
[Route to downstream automation]
Enter fullscreen mode Exit fullscreen mode

Push to Incident Tracker

Currently reports live in Slack only. For trend tracking (which failure category appears most this month, which service generates the most errors):

[Post to Slack]
       ↓
[Jira: Create Issue] OR [Linear: Create Issue]
Fields: title=error title, label=category, priority=severity
Enter fullscreen mode Exit fullscreen mode

Alternative Keyword Sources

Sentry is one input, not the only one. Any system that can POST a JSON body to the webhook URL — a custom logging pipeline, a different error tracker, CloudWatch alerts, a Datadog webhook — can feed this workflow.

Limitations

Classification quality depends on logging quality. A stack trace with a real culprit and clear exception type gives the model solid evidence. A log entry that says Exception: failed gives it nothing — the workflow correctly labels that uncertain rather than guessing. Better logging matters more than classifier tuning for sparse errors.

No persistent storage for trend analysis. The dedup fingerprint store uses n8n workflow static data, which is in-memory and resets on workflow restart. For cross-session dedup or trend reporting, connect to a database or sheet instead.

The uncertain category is a feature. If you find everything is coming back uncertain, the problem is upstream logging quality, not the classifier threshold.

Get the Free Workflow JSON

IT Path Solutions published the complete n8n workflow — webhook trigger, Extract & Dedup node, Gemini classification chain, Structured Output Parser, fix playbook mapping, Slack Block Kit reporting, and error handling — all pre-connected.

Import into any n8n instance, add credentials, send a test error through the webhook.

👉 Download the free n8n AI agent monitoring workflow

Setup guide covers: Gemini credential config, Slack bot token scoping, Sentry webhook setup, dedup window tuning, and how to test end-to-end before pointing production traffic at it.

Summary

The pipeline works because it does the repetitive diagnostic work — normalisation, dedup, classification, playbook lookup — before a human sees the alert. The on-call engineer opens Slack and sees a structured triage report instead of a raw stack trace to interpret.

The honest uncertain fallback is what makes it trustworthy: it admits when it doesn't know rather than sending an engineer down the wrong path with a confident wrong answer.

Run one test error through the webhook to confirm the chain works before enabling for production traffic.

Full guide and JSON: itpathsolutions.com/ai-agent-monitoring-workflow

Running something similar in production? What failure category shows up most? Drop it in the comments.

Top comments (0)