When you ask a language model why a service broke it will tell you. Confidently, fluently, and with assurance, every time, regardless of whether the evidence supports the conclusion it reached or not.
For a post-incident review, that is worse than useless. A plausible answer with nothing behind it still gets written into the postmortem, and somebody WILL act on it.
Because of this issue, I built the chain the other way around. It is computed first, deterministically, and a model is only allowed to narrate what has already been established. Below is the design, the specific numbers it runs on, and what it prints.
Three sources that never mention each other
It is the morning after an incident. Your checkout API began failing readiness probes at 03:20AM and as the DevOps, you need a timeline in order to create the review.
Noticing the outage was never the difficult part. Reassembling it is:
- CloudTrail records that somebody called an AWS API, and when.
- AWS Config records what a resource looked like before and after a change.
- The Kubernetes API records that a pod started failing its readiness probe.
None of those payloads reference the others. CloudTrail names a security group. The Kubernetes event names a pod. Neither one knows the other exists, and AWS Config publishes relationships only between AWS resources, so it will never mention a pod either.
Assembling the chain by hand takes about an hour of moving between consoles and comparing timestamps. The result is as reliable as how awake you were while doing it.
The obvious shortcut is to feed all three outputs into a language model and ask "what happened?". It will answer. Regardless of the amount of data it gathers, it will stop "investigating" when it thinks it found the answer. That is precisely the problem.
Ordering is only meaningful past the noise floor
Every source has a different timestamp precision, so "A happened before B" is a claim you can only make once the gap between them exceeds the combined uncertainty of both. The engine carries an explicit floor per source:
| Source | Timestamp uncertainty |
|---|---|
| Kubernetes events | 5s |
| Kubernetes deployments | 30s |
| CloudTrail | 60s |
| AWS Config | 600s |
AWS Config is the coarse one because the recorder samples state after the fact rather than at the moment of change. These figures are deliberately conservative. Overstating precision manufactures false causality, whereas overstating imprecision only makes the engine say it cannot tell, which is the failure worth preferring.
A CloudTrail change and a Kubernetes event therefore need to be more than 65 seconds apart before their ordering counts for anything. Inside that margin the link is still reported, but never as anything better than weak, with the reason stated plainly: ordering cannot be established from timestamps alone.
Two further rules follow from the same logic:
- A change that occurred after the symptom is never proposed as its cause. This is the rule a model asked to find a root cause will cheerfully break.
- A change more than one hour before the symptom is not proposed either. Long-fuse failures are real, a certificate expiring or a disk slowly filling, but quietly entertaining a twenty-hour-old change as the cause of a sudden failure produces confident nonsense far more often than insight. It is recorded as a gap instead.
Connection comes from declared fields, not resemblance
Ordering alone is not causation, so the engine separately tries to establish that the two events concern connected resources. It works strongest first:
- Same resource. Both events name the same thing.
- Declared relationship. One side explicitly lists the other, via an AWS Config relationship or a Kubernetes owner reference.
- Resolved topology. A multi-hop path of authoritative fields connects them: a security group to a network interface, to an instance, to a node, to a pod. Each hop is citable.
- Same scope. Both happened in the same namespace. Weak, and labelled as inferred.
Only the first two produce a Strong link. Multi-hop topology and shared scope produce Moderate. Ordering with no established connection produces Weak.
That third case is worth dwelling on, because it is where most tools would overclaim. A security group really is attached to the node your failing pod runs on, and the path is real and fully cited. But every pod on that node shares those security groups, so the path proves the change could have reached this symptom, not that it did. It is capped at Moderate permanently. Confidence is reported as a band rather than a percentage, because a precise-looking number would imply a calibration nobody has earned.
What it prints
You give it a symptom in your own words and a window:
kubeopsai-reconstruct \
--symptom "checkout-api started failing readiness probes" \
--namespace prod --window 60 \
--aws AWS::EC2::SecurityGroup=sg-0abc123
The output is Markdown that drops into a postmortem template unmodified. Abridged to the parts that matter:
Summary
Candidate cause (moderate confidence): alice@corp called RevokeSecurityGroupIngress on sg-0abc123 at 2026-02-14 03:05:00 UTC.
Attributed to:
alice@corpEvidence:
cloudtrail:a1b2c3d4Why moderate: the resources are connected by recorded relationships (listed below), but that path is shared by everything on the same node, so it shows the change could have reached this symptom rather than that it did.
Chain of events
Moderate: change preceded the symptom by 900s, beyond the 65s combined timestamp uncertainty; the resources are connected by 4 recorded relationships: sg-0abc123 to eni-04d1, to i-0abc123def456, to ip-10-0-1-5, to checkout-api-7d9f
What could not be established
metric: this reconstruction did not query when resource usage deviated, so nothing from that source could contribute.
To close this gap: pointKUBEOPSAI_PROMETHEUS_URLat a reachable Prometheus.Source coverage
Source Outcome cloudtrail 1 event k8s_event 1 event aws_config empty
Four things in that output carry the entire design.
It refuses to promote a moderate finding. The heading says candidate, not cause, and a dedicated line explains exactly why the confidence is capped. Nothing about the phrasing invites you to skim past the caveat.
It names a person. alice@corp, cited to a specific CloudTrail EventId. Not "a configuration change occurred" but who made it, verifiable against the raw event in your own account.
It states what it could not see. That section is not an apology, it is what makes everything above it credible. A chain that conceals its holes is worse than no chain, and every blocking gap also names the thing to instrument in order to close it. Note that the coverage table distinguishes a source that was queried and came back empty from one that was never queried at all. Those are different facts, and collapsing them would imply the tool looked somewhere it did not.
Every claim carries a citation. Each reference resolves on its own. The evidence appendix detaches cleanly and can go to an auditor untouched.
When there is nothing to report, it reports nothing
If no change in the window can be connected to the symptom, the summary says so:
No cause was established. Either the cause lies outside this window, or outside the data sources available.
No narrative gets constructed to fill the space. A security group revocation with no established path to the failing pod is listed under "Changes with no linked symptom" instead, surfaced for your judgement rather than asserted as the answer.
The engine also keeps several candidates rather than collapsing to one, deliberately. Presenting a single change as the cause is the specific overreach the whole design exists to avoid.
Where the model still earns its keep
None of this makes the language model useless. It keeps it in its lane.
Once the facts are fixed, a model is genuinely good at reading a chain, explaining it in plain prose, and ranking which of several candidates deserves attention first. What it cannot do is introduce a link the deterministic pass did not establish, and every citation it writes is checked back against collected evidence before the report ships.
Given the same events, the engine returns the same links every time. No approach built on asking a model why can offer that.
The collectors are free
The 23 collectors this reads from, covering Loki, Prometheus, Tempo, the Kubernetes API, CloudTrail, AWS Config, ECR and RDS, are Apache 2.0 and free to use as an MCP server. Point Claude, Amazon Q, Kiro or Cursor at your cluster and inspect exactly what gets collected before committing to anything:
Free MCP server: github.com/kubeopsai/k8s-telemetry-mcp
The reconstruction engine itself, meaning the deterministic linker, the gap accounting and the evidence report, is the commercial piece: kubeopsai.net
I would genuinely like to be argued with on the linking rules, specifically about cases where timestamp precedence plus declared relationships would produce a link that is wrong. That is the failure mode I most want to find.
Top comments (0)