Chaining AI Assistants for Incident Investigation: From Grafana to an RCA Everyone Can Use

Our monitoring dashboard insisted that everything was fine. Across an entire week and every single namespace in a production Kubernetes platform, we recorded exactly eight container restarts.

Meanwhile, customer reports kept piling in: live calls were abruptly cutting off.

Both statements were completely accurate. The dashboard was simply answering a different question from the one our users were experiencing. Bridging that disconnect required chaining four specialized AI assistants, asking precise forensic questions, and sticking to one non-negotiable rule: nothing an assistant says counts as evidence until it is backed by an executable query, a raw log line, or a verifiable platform event.

Here is the investigation workflow, the prompts that surfaced the real data, and how to structure a post-incident Root Cause Analysis (RCA) that technical leads, support teams, and auditors can actually rely on.

---

Each Assistant Sees Only One Slice of the Failure

The single biggest mistake in AI-assisted troubleshooting is asking a single model to diagnose an entire distributed system.

• Telemetry reveals the effect.
• The cloud control plane reveals the external trigger.
• The codebase reveals intended design.

No single tool or context window has end-to-end visibility across all three.

Layer	Core Question	Assistant / Tooling	Inherent Blind Spot
Telemetry (metrics, logs, traces)	What broke, when did it happen, and how often?	Grafana Assistant (Investigation Mode)	Only sees connected data sources and what your RBAC token permits.
Cloud Control Plane	Why did the underlying platform take action (drain, upgrade, scale)?	Azure Copilot / CLI in the cloud portal	Understands infrastructure events, but is completely blind to internal application state.
Database	Did queries stall, or did a failover trigger?	MongoDB Atlas Performance Advisor / Query Profiler / MCP Server	MCP tooling typically samples recent slow queries; deeper patterns require direct profiling UI.
Code & Config	How does the service handle SIGTERM, timeouts, and connection draining?	Claude Code / GitHub Copilot (Read-Only)	Reading source code shows intent; it never proves what production actually did.

If your observability signal lives on Datadog, Honeycomb, or New Relic, the workflow remains identical: start with telemetry, isolate the platform boundary, then inspect the application contract.

---

The Investigation Loop

1. Frame the Problem Statement Before Opening Any Tool

An investigation is strictly bounded by the precision of its initial problem statement. I use this structured template every time:

Incident summary: <what users or the platform report, 1-2 sentences>
Timeframe: <start UTC> to <end UTC> (include first user symptom and last known good state)
Scope: <clusters, namespaces, services, cloud providers>
Current dashboard signal: <e.g., container restart count across namespaces>
What does not match: <e.g., users report dropped audio streams, but pod restarts remain near zero>
Focus areas: <pod lifecycle, node events, provider timeouts, latency spikes, deploy history>
Constraints: <which logs are ingested vs. which signals are uncollected>
Output requested: UTC timeline, hypothesis list with status, confidence level, evidence gaps, next steps

The critical line is "What does not match." It forces you to define the exact contradiction. It leads directly to the question that exposes dashboard blind spots: If the user report is true, what metric would show it, and why aren't we tracking it?

2. Run a Hypotheses-Driven Investigation

In Grafana Cloud, switch Assistant Workspace to Investigation mode and supply your structured problem statement.

A dedicated investigation tool queries metrics, logs, and traces while managing a living hypothesis tree (open, root cause, symptom, disproven, or blocked). Crucially, every finding links back to the underlying PromQL or LogQL query.

Best practices for this step:

• Drive it iteratively with /ask: Ask targeted side questions without resetting the entire context.
• Actively disprove weak hypotheses: LLMs tend to anchor on the first plausible explanation. If a theory doesn't hold up under load metrics, explicitly mark it as disproven.
• Respect permission boundaries: Tooling can only query what your service account can access.

3. Turn Evidence Gaps into Targeted Platform Questions

A solid initial investigation concludes with an explicit Evidence Gaps list. Resist the urge to paper over these gaps with assumptions. Instead, route each gap to the platform tool that owns that domain:

Evidence Gap	Source of Truth
Why did a node drain or upgrade cycle fail?	Cloud Control Plane (Azure Activity Log / AKS events)
Why are eviction events missing from metrics?	Observability team (Missing Prometheus scrape configs)
Does the application gracefully close WebSockets on termination?	Codebase & Helm charts (preStop, terminationGracePeriodSeconds)
Did database latency spike during the eviction?	Database profiling / Atlas Performance Advisor

4. Query the Control Plane with Exact Timestamps and Demand Raw Data

Vague prompts generate conversational hallucinations. Feed exact timestamps, cluster IDs, and resource groups directly into your platform assistant:

For AKS cluster <cluster-name> in resource group <rg-name>:
1. What is the provisioning state of node pool <pool-name> and when did it last change?
2. List upgrade, scale, and drain operations between <start UTC> and <end UTC>,
   including trigger source (user, auto-upgrade, maintenance schedule).
3. For the failed operation at <timestamp UTC>, provide the exact error code, 
   error message, and the specific pod or PodDisruptionBudget that blocked the drain.
4. Output raw Activity Log and Resource Health events for this window.
Do not provide a high-level summary. Return exact logs with timestamps and operation IDs.

An AI summary is never incident evidence. Always extract and preserve the raw event payload.

5. Feed Raw Findings Back to Update the Hypotheses

Instead of manually reconciling different chat windows, feed the platform output back into the primary telemetry assistant:

New raw evidence from Azure Control Plane (timestamps in UTC):
<paste raw log/event entries>

Task: Update the active investigation. Re-evaluate each hypothesis status, 
update confidence ratings, close resolved evidence gaps, and list remaining unknowns.
Preserve all earlier evidence. Cite this data source as Azure Activity Log.

Avoid summarizing findings between assistants. Summaries strip out vital nuance while artificially inflating certainty. After two hops, you are debugging an AI game of telephone.

6. Inspect Application Code (Read-Only)

Once platform metrics and control plane events align, bring in a coding agent (such as Claude Code or GitHub Copilot) via read-only MCP servers.

For example, connect the Grafana MCP server with strict read-only flags:

claude mcp add-json "grafana" '{
  "command": "mcp-grafana",
  "args": ["--disable-write"],
  "env": {
    "GRAFANA_URL": "<your-grafana-url>",
    "GRAFANA_SERVICE_ACCOUNT_TOKEN": "<viewer-level-token>"
  }
}'

Formulate investigative questions rather than asking for code modifications:

Read-only investigation. Do not modify files.
1. Inspect how <service> handles SIGTERM. Is there an active preStop hook or custom terminationGracePeriodSeconds?
2. Map container image tag <tag> to its commit and inspect changes affecting connection draining.
3. Locate where log string "<exact error message>" is defined and what conditional branches trigger it.
4. For each proposed mitigation, identify the exact YAML or source files that would need adjustments.
Cite exact repository paths and line numbers for all assertions.

Code inspection only proves intent. Any conclusion derived solely from code remains classified as inferred until production logs confirm it.

7. Red-Team the Root Cause

Before finalizing your conclusions, challenge your findings with a fresh, isolated session:

Act as a skeptical Principal SRE reviewing this incident RCA draft. 
Identify the three strongest alternative explanations for these symptoms, 
highlight any claims stated with higher confidence than the raw evidence supports, 
and point out any remaining data gaps.

Designing an RCA for Multiple Audiences

An RCA that can only be understood by the engineer who wrote it fails to deliver value to the organization.

Audience	What They Need	Where It Belongs
Engineering Leadership & Product	What broke, user impact, and whether it can happen again.	Executive Summary & Impact Analysis
On-Call SREs & Engineers	Detailed chronology to correlate with past alert fatigue.	UTC Timeline with citations per event
Platform Maintainers	Verifiable data, rejected hypotheses, exact changes needed.	Evidence, Disproven Theories, Remediation Matrix
Auditors & Compliance	Proof of due diligence and transparency on unknowns.	Confidence tiers, Evidence Gaps, Verification tests

Best Practices for Defensible Post-Mortems

1. Lead with a 3-Sentence Executive Summary: State the failure, the root cause, and current remediation status.
2. Explicitly Label Every Statement:
   • Observed: Directly confirmed by telemetry or log lines.
   • Inferred: Architecturally consistent with telemetry, but lacks direct trace verification.
   • Unverified: Relies on third-party report; pending forensic validation.
3. Document Disproven Hypotheses: Documenting what didn't happen prevents future responders from wasting cycles on dead ends.
4. Make Action Items Concrete: Replace "Add PDB alerts" with "Deploy PDB Prometheus exporter rules; assign to @infra; verify by testing a cordon on staging by End of Sprint."
5. Close the Loop with Verification: An incident is not resolved when the post-mortem is merged. It is resolved when the queries that detected the original anomaly run clean post-deployment.

Production Guardrails for AI Investigation

• Read-Only by Design: AI assistants should query and synthesize. Only human engineers approve changes through standard GitOps workflows.
• Re-run Critical Queries Manually: Never accept an AI-generated conclusion without manually running the 2 or 3 underlying database or PromQL queries yourself.
• Strict Secret Sanitization: Never paste API tokens, connection strings, or customer PII into prompts.
• Assistants Are Not Sources: Never write "The assistant found that..." in an RCA. Cite the log line, PromQL metric, or cloud activity log entry.

If you have a different assistant workflow or run incident investigations across multi-cloud environments, share your approach in the comments below.