Incident Context: tracing production failures without guessing

This is a submission for the Sanity Challenge, Path One: Ship an Agent That Queries Real Content.

What I Built

Incident Context is an incident-investigation agent for questions that cannot afford a confident but unsupported answer.

During an outage, evidence is rarely contained in one document. It is spread across services and dependencies, deployments and released versions, configuration changes, previous incidents, and version-specific runbooks.

Incident Context models those records as connected Sanity documents. The agent follows their relationships through a Sanity Context MCP Knowledge Base, separates confirmed evidence from inference, and preserves the source paths behind every report.

The Problem

A keyword search can find "timeout" in both INC-142 and INC-208. But correlation is not causation.

Both incidents were production outages on consecutive days. Both involved configuration changes. Both mentioned timeouts. That is not enough to determine whether they share a root cause or require the same remediation.

The useful answer is not a matching paragraph. It is a supported path across an incident, an affected service, a deployment, a configuration change, and an applicable runbook.

The Solution

The agent models operational records as a connected graph:

Service    → depends on → Service
Deployment → belongs to → Service
Deployment → includes   → Change
Change     → modifies   → Service
Runbook    → applies to → Service
Incident   → affects    → Service
Incident   → relates to → Deployment
Incident   → relates to → Change
Incident   → references → Runbook

When you ask it to compare the two incidents, the agent follows the structured path, not keyword matches:

INC-208
→ affected checkout-api
→ related deployment checkout-api v2.3.0
→ included PAYMENT_TIMEOUT_MS configuration change
→ reduced from 5000ms to 1500ms

This traversal becomes an evidence trail in the interface:

Notice the evidence trail at the top. It shows the relationships the agent followed: INC-142 affected payment-api and was caused by a DB_POOL_SIZE reduction; INC-208 affected checkout-api and was caused by a PAYMENT_TIMEOUT_MS reduction. Two different paths. Two different root causes.

Below that: 4 confirmed evidence items, 0 inferences, 2 sources. The model did not guess or correlate. It followed the graph.

Why This Matters

Operational data contains many correlations:

• a deployment happened before an incident
• two incidents mention timeouts
• a runbook belongs to an affected service
• a configuration value changed near an outage

Those relationships are useful evidence. They are not automatically proof of causation.

The agent preserves the distinction. Every investigation returns:

1. A direct answer — plain language, grounded in the evidence
2. An evidence trail — the relationship path the agent followed
3. Confirmed evidence — facts from the Knowledge Base
4. Inferences — what the model had to infer (or in this case, nothing)
5. A recommended next step — actionable based on what was discovered
6. Sources — exact Sanity paths for verification

If Sanity Context cannot be reached, the application does not generate an unsupported fallback answer.

How I Used Sanity

Sanity is not being used as a generic document store behind a chat box. Its references form the investigation graph.

The project defines five document types: service, deployment, change, runbook, incident. Their references encode the operational relationships.

During the build, I included the references required for an investigation:

• service.dependencies[] → service
• deployment.service → service
• deployment.changes[] → change
• change.service → service
• runbook.service → service
• incident.affectedServices[] → service
• incident.relatedDeployments[] → deployment
• incident.relatedChanges[] → change
• incident.relatedRunbook → runbook

Sanity Context distilled those connected documents into navigable, source-linked entries. At investigation time, the agent reads the Knowledge Base outline through initial_context, selects relevant entries, retrieves them through Knowledge Base tools, and traces explicit relationships. The system prompt explicitly forbids turning correlation into confirmed causation.

The Demo

Live application: https://incident-context.vercel.app/

No login is required.

The dataset represents a small production system:

web-app → checkout-api → payment-api → postgres

Try these questions:

What changed before INC-208?
Trace the services affected by INC-208.
Which runbook applies to INC-142?
Compare INC-142 and INC-208.

The comparison question is the strongest demonstration. It asks the agent to distinguish incidents that share related terminology but have different evidence paths.

Code

Repository: https://github.com/anilloutombam/incident-context

The repository contains two standalone applications:

agent/   Next.js investigation interface and API
sanity/  Sanity Studio, schemas, and demo data

Runtime Architecture

flowchart LR
    User[Incident question] --> UI[Next.js interface]
    UI -->|POST /api/chat| API[Next.js API]
    API --> Loop[AI SDK agent loop]
    Loop <-->|model requests| Gemini[Gemini 3.5 Flash-Lite]
    Loop <-->|MCP tool calls| MCP[Sanity Context MCP]

    Content[Sanity Content Lake<br/>Services · Deployments · Changes<br/>Runbooks · Incidents]
    Content -->|builds| KB[Sanity Knowledge Base]
    KB -->|serves cited entries| MCP

    Loop --> Report[Zod-validated report]
    Report -->|JSON response| UI
    UI --> Output[Evidence trail · Confirmed evidence<br/>Inferences · Next step · Sources]

The Gemini key and Sanity organization token remain on the server. The browser receives a validated report, not credentials.

Structured Output

The API validates the final response against a Zod schema instead of asking the model to return arbitrary Markdown:

{
  answer,
  evidenceTrail: [
    {
      from,
      relationship,
      to,
      source
    }
  ],
  confirmedEvidence,
  inferences,
  recommendedNextStep,
  sources: [
    {
      label,
      path
    }
  ]
}

This allows the interface to render relationships, facts, inferences, and sources as distinct product elements instead of parsing presentation from a generated string.

Production Safeguards

The deployed agent includes:

• per-client request rate limiting
• a 500-character question limit
• a 30-second request timeout
• explicit Gemini, Sanity, timeout, and rate-limit errors
• no unsupported answer when the Knowledge Base is unavailable
• a health endpoint at /api/health
• server-only Gemini and Sanity credentials

Testing the MCP Dependency

The project uses the published mcp-failure-lab package to test the dependency the agent cannot operate without.

The live compatibility scenario calls initial_context and verifies that the Sanity Context endpoint responds within ten seconds.

A separate local resilience suite covers bounded delays, hanging requests, malformed MCP responses, and connection loss.

pnpm test:mcp
pnpm test:mcp:live
pnpm test:mcp:faults

What I Learned

The difficult part was not connecting a model to an MCP endpoint. It was deciding what the model was allowed to claim.

I initially let the model connect any dots it found—deployments before incidents, shared keywords, related services. The output looked authoritative. But INC-142 and INC-208 both mentioned timeouts and both affected the payment path. One was a database connection exhaustion issue. One was a configuration change that had been too aggressive. The model could see both, but it could not reliably say which caused which.

I had to split facts from guesses in the output, enforce it in the system prompt, and then show that separation in the UI. Because incident responders need to know what you are certain about.

The useful answer is not a matching paragraph. It is a supported path, with source citations, and an explicit boundary between evidence and inference.

Comments

[mthburnsbarber-web]

The confirmed evidence vs. inference separation is the right design decision for incident investigation tools. The INC-142 / INC-208 comparison case makes it concrete — both mentioned timeouts, both were in the payment path, but one was DB connection exhaustion and one was an aggressive PAYMENT_TIMEOUT_MS reduction. A system that collapses those into "similar incidents" because they share keywords would actively mislead an on-call engineer.

"I initially let the model connect any dots it found. The output looked authoritative" — this is the failure mode that's easy to miss because authoritative-looking output is hard to distinguish from correct output without knowing the ground truth. The split between confirmed evidence (4 items) and inferences (0 items) being surfaced in the UI is the right call; it forces the model's epistemic state to be visible rather than hidden in the prose.

Modeling operational records as a connected graph (Service → depends on → Service, Incident → relates to → Deployment, etc.) rather than a flat document store is what enables path-following rather than keyword matching. The structured Zod output schema — evidence trail, confirmed evidence, inferences, next step, sources as separate fields — makes that path visible in the interface rather than buried in a paragraph.

The mcp-failure-lab dependency testing is a nice touch. Testing what happens when the Knowledge Base is unavailable is the thing most agent builds skip.