Context Engineering for On-Call AI Agents: What to Feed an Incident Agent

💡 Originally published on devtocash.com — where this guide stays updated. I write hands-on DevOps/SRE deep-dives there weekly.

The window is the product

An on-call AI agent doesn't fail because the model is dumb. It fails because you handed it the wrong 8,000 tokens. Give an incident agent a raw alert payload and a kubectl tool and it will happily investigate — badly — because it doesn't know that payment-service was deployed four minutes ago, that this exact 5xx pattern was a bad config map last Tuesday, or that the runbook says "check the Redis connection pool first." Context engineering is the discipline of assembling exactly the right facts into the model's window on each turn, and it is the single highest-leverage thing you control after the harness itself.

This is a practitioner's guide to doing it for an incident-response agent: what to retrieve, how to budget the window, how to keep it from rotting, and how to write lessons back so the agent gets better after every page.

Context is a budget, not a bucket

The instinct is to dump everything: full logs, every metric, the whole runbook, last month's incidents. That is exactly wrong. A bloated window degrades reasoning — the model loses the thread in a wall of low-signal text (the "context rot" problem), latency climbs, and cost per investigation balloons. Treat the window like a capacity plan with a fixed budget:

Slot	Budget	Content	Source
System / role	~800 tok	Who the agent is, its guardrails, output schema	Static prompt
Alert	~400 tok	Parsed alert: service, severity, firing metric	Alertmanager webhook
Recent changes	~1,000 tok	Deploys + config changes in the last 60 min	CI/CD + Git
Live telemetry	~2,500 tok	Top anomalous metrics, error-log sample, pod events	Prometheus / Loki
Retrieved runbook	~1,500 tok	The one relevant runbook section, not the whole doc	Vector store
Similar past incidents	~1,200 tok	1–2 nearest postmortems with resolution	Memory store
Working scratch	remainder	The agent's own notes across turns	Loop state

The numbers matter less than the principle: every slot is capped, and something has to be evicted to add more. When you find yourself wanting a bigger window, the real problem is almost always weak retrieval, not too little space.

Retrieve, don't stuff

The runbook slot is where most teams go wrong. A full incident runbook can be 3,000+ words; pasting all of it buries the relevant paragraph. Instead, chunk your runbooks and postmortems, embed them, and retrieve only the sections that match the live incident.

def assemble_context(alert, budget):
    query = f"{alert.service} {alert.metric} {alert.summary}"

    # Retrieve the SECTIONS that match this incident, not whole docs
    runbook = vector_store.search(query, collection="runbooks", k=2)
    similar = vector_store.search(query, collection="postmortems", k=2)

    # Fresh, structured signals — always cheaper than raw logs
    deploys = git_api.deploys_since(alert.service, minutes=60)
    metrics = prom.top_anomalies(alert.service, window="15m", n=5)
    logs    = loki.error_sample(alert.service, window="10m", limit=20)

    return trim_to_budget({
        "alert": alert.as_summary(),
        "recent_changes": deploys,
        "telemetry": {"metrics": metrics, "logs": logs},
        "runbook": [r.text for r in runbook],
        "past_incidents": [s.resolution for s in similar],
    }, budget)

Two rules make this reliable. First, retrieve structured, not raw: prom.top_anomalies() returning five ranked series beats dumping 200 metric lines; a 20-line error sample beats 10 MB of logs. The agent reads real telemetry the way an autonomous SRE agent does — pre-digested into ranked, labeled facts. Second, the tools are the escape hatch: don't pre-load everything you might need. Load the high-probability context, and expose a scoped MCP server so the agent can pull more on demand — a deeper log query, a specific trace — only when its reasoning calls for it.

Recency beats completeness

For an incident, freshness dominates. A deploy from four minutes ago is worth more than a perfectly relevant runbook written a year ago. Weight retrieval accordingly — decay the score of old postmortems, and always place the "recent changes" slot near the top of the window where the model attends most strongly:

def score(doc, query_sim):
    age_hours = (now - doc.timestamp).total_seconds() / 3600
    recency = math.exp(-age_hours / 168)   # ~1 week half-life
    return 0.6 * query_sim + 0.4 * recency

Order inside the window is itself context engineering. Models attend most to the beginning and end of a long context (the "lost in the middle" effect), so put the alert and recent changes first, the agent's output instructions last, and the bulkier retrieved passages in the middle where a little dilution is acceptable.

Memory: close the loop

A stateless agent solves the same incident from scratch every time. The payoff comes when each investigation writes a durable lesson back — turning the on-call loop into a system that compounds, the way a persistent memory layer upgrades any agent. After resolution, store a compact, structured record:

{
  "service": "payment-service",
  "signature": "5xx spike + p99 latency > 2s within 5m of deploy",
  "root_cause": "connection pool exhausted after maxConns lowered in cm/redis",
  "resolution": "kubectl rollout undo deploy/payment-service",
  "confirmed_by": "human",
  "ts": "2026-07-21T03:14:00Z"
}

Discipline matters more than volume. Write signatures and resolutions, not chat transcripts — those are what future retrieval matches against. Only persist human-confirmed outcomes; an agent that memorializes its own unverified guesses will confidently retrieve last week's hallucination into this week's incident. Prune aggressively, and let the same recency decay demote stale entries so a fixed-long-ago cause doesn't keep resurfacing.

Failure modes to design against

Context engineering has its own reliability hazards. Design for them explicitly:

• Context rot. Past ~50% of the model's usable window, quality drops before the hard limit does. Cap the budget well under the max and measure — don't assume a 200K window means you should fill 200K.
• Poisoned memory. One wrong "confirmed" root cause retrieved as authoritative can misdirect every similar future incident. Gate writes behind human confirmation and keep memory entries editable and revocable.
• Stale runbooks. Retrieval is only as good as the corpus. A runbook referencing a deleted service is worse than none — it sends the agent chasing ghosts. Re-index when docs change.
• Silent truncation. When trim_to_budget drops a slot, log it. An agent that silently lost its telemetry slot reads as confident but is flying blind — the same trap as unobservable automation.

Measure it like any pipeline

You can't tune what you don't observe. Trace every investigation with OpenTelemetry: record which context slots were filled, how many tokens each consumed, which retrieved chunks the agent actually cited, and the final outcome. Two metrics tell you whether your context engineering is working — retrieval hit rate (did the relevant runbook/postmortem make it into the window?) and grounded-resolution rate (did the agent's conclusion cite retrieved context, or invent one?). Feed both into evals you run before the agent touches prod, replaying past incidents to check the right context surfaces for each.

Takeaway

An on-call agent's intelligence lives in its context window, and that window is something you engineer, not something you fill. Budget every slot, retrieve structured facts instead of stuffing raw data, weight recency for live incidents, write only confirmed lessons back to memory, and trace the whole thing so you can tune it. Get the context right and a modest model runs a competent investigation; get it wrong and the best model on the market will confidently investigate the wrong thing.

---

📌 Read the latest version of this guide — plus the full library of DevOps, SRE, Kubernetes, observability & cloud-cost guides — on devtocash.com.