We're going to build a structured root cause analysis agent that reads an incident description and performs explicit multi-step reasoning before concluding. It breaks problems into hypotheses, tests them against evidence, and ranks likelihood. This kind of deep reasoning reduces false positives in operational triage and saves engineering time.
What you'll need
- Python 3.10 or newer
- The OpenAI SDK:
pip install openai - An Oxlo.ai API key from https://portal.oxlo.ai
I also recommend setting OXLO_API_KEY as an environment variable so you do not hardcode secrets.
Step 1: Set up the Oxlo.ai client
The Oxlo.ai API is fully OpenAI-compatible, so the client is a drop-in replacement. Point the base URL to Oxlo.ai and load your key.
from openai import OpenAI
import os
client = OpenAI(
base_url="https://api.oxlo.ai/v1",
api_key=os.environ["OXLO_API_KEY"]
)Step 2: Define the incident and reasoning schema
To force deep reasoning instead of shallow guessing, we lock the output into a JSON schema. The model must decompose the problem, list competing hypotheses with evidence for and against each, and then commit to a conclusion. Here is the incident we will analyze and the schema.
import json
INCIDENT_REPORT = """
Service: payment-gateway
Time: 2024-05-17 14:03 UTC
Symptoms: Error rate spiked from 0.1% to 12%. Latency p99 jumped from 120ms to 4s.
Only affecting checkout flow. Database CPU normal. Cache hit rate dropped to 15%
from 85%. No deployments in last 4 hours. Third-party fraud check API timing out.
"""
REASONING_SCHEMA = {
"type": "object",
"properties": {
"problem_decomposition": {
"type": "array",
"items": {"type": "string"}
},
"hypotheses": {
"type": "array",
"items": {
"type": "object",
"properties": {
"hypothesis": {"type": "string"},
"evidence_for": {
"type": "array",
"items": {"type": "string"}
},
"evidence_against": {
"type": "array",
"items": {"type": "string"}
},
"confidence": {
"type": "string",
"enum": ["high", "medium", "low"]
}
},
"required": ["hypothesis", "evidence_for", "evidence_against", "confidence"]
}
},
"most_likely_cause": {"type": "string"},
"recommended_fix": {"type": "string"},
"confidence_score": {"type": "integer", "minimum": 1, "maximum": 10}
},
"required": ["problem_decomposition", "hypotheses", "most_likely_cause", "recommended_fix", "confidence_score"]
}Step 3: Build the system prompt for deep reasoning
The system prompt is the most important part of this build. It explicitly instructs the model to reason before concluding, consider competing explanations, and avoid anchoring on the first idea. Keep this prompt editable. You can tune the rigor without touching code.
SYSTEM_PROMPT = """You are a senior site reliability engineer performing root cause analysis.
Your task is to reason deeply about an incident before concluding.
Follow these rules:
1. Decompose the incident into distinct sub-problems.
2. Generate at least three competing hypotheses.
3. For each hypothesis, list concrete evidence for and against it. Do not cherry-pick.
4. Compare hypotheses explicitly and select the most likely cause.
5. Recommend a specific, actionable fix.
6. Respond ONLY as a JSON object matching the provided schema."""Step 4: Generate the initial analysis
We call Kimi K2.6, which handles advanced reasoning and agentic coding tasks well. We use JSON mode so the model is constrained to our schema. Oxlo.ai serves this with no cold starts, and because pricing is flat per request, the cost does not balloon when the incident report or reasoning trace grows long.
def generate_rca(incident_text):
user_message = (
f"Incident report:\n{incident_text}\n\n"
f"Respond with JSON following this schema:\n{json.dumps(REASONING_SCHEMA, indent=2)}"
)
response = client.chat.completions.create(
model="kimi-k2.6",
messages=[
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": user_message},
],
response_format={"type": "json_object"},
temperature=0.2,
)
return json.loads(response.choices[0].message.content)
initial_rca = generate_rca(INCIDENT_REPORT)
print(json.dumps(initial_rca, indent=2))Step 5: Critique and refine with a second model
Deep reasoning improves when a second party stress-tests the logic. We pass the draft RCA to Llama 3.3 70B and ask it to find gaps. Running both models on Oxlo.ai keeps the stack simple, and adding this critique pass does not scale costs with token count.
CRITIC_PROMPT = """You are a skeptical peer reviewer. Review the following root cause analysis.
Identify logical gaps, missing evidence, or alternative explanations that were not considered.
Output your critique as a JSON object with fields:
- gaps (array of strings)
- missed_hypotheses (array of strings)
- overall_assessment (string)"""
def critique_rca(rca):
payload = json.dumps(rca, indent=2)
response = client.chat.completions.create(
model="llama-3.3-70b",
messages=[
{"role": "system", "content": CRITIC_PROMPT},
{"role": "user", "content": f"RCA to review:\n{payload}"},
],
response_format={"type": "json_object"},
temperature=0.3,
)
return json.loads(response.choices[0].message.content)
critique = critique_rca(initial_rca)
print(json.dumps(critique, indent=2))Run it
Putting it together, the full script loads the incident, generates a structured RCA, and then critiques it. This is the exact flow I run in my own triage toolkit.
if __name__ == "__main__":
print("=== Initial Deep Reasoning ===")
rca = generate_rca(INCIDENT_REPORT)
print(json.dumps(rca, indent=2))
print("\n=== Peer Review ===")
review = critique_rca(rca)
print(json.dumps(review, indent=2))When I run this against the sample incident, the output looks like this:
=== Initial Deep Reasoning ===
{
"problem_decomposition": [
"Why did error rates spike only in checkout?",
"Why did cache hit rate collapse?",
"Why did latency degrade despite normal database CPU?"
],
"hypotheses": [
{
"hypothesis": "The third-party fraud API is timing out and blocking checkout completion.",
"evidence_for": [
"Symptoms mention the fraud check API is timing out.",
"Only checkout is affected, which relies on fraud checks.",
"Latency increased dramatically, consistent with blocking external calls."
],
"evidence_against": [
"If the API simply timed out, we might expect a smaller error rate spike if timeouts are handled gracefully.",
"Cache hit rate dropping is not directly explained by an external API timeout."
],
"confidence": "medium"
},
{
"hypothesis": "A cache invalidation bug is causing cache misses and thundering herd to the fraud API.",
"evidence_for": [
"Cache hit rate dropped from 85% to 15%.",
"Database CPU is normal, so load is not hitting the DB.",
"Latency spike is consistent with repeated cache misses and repeated fraud API calls."
],
"evidence_against": [
"No deployment occurred in the last 4 hours, so code did not change.",
"Cache invalidation usually follows a deployment or config push."
],
"confidence": "high"
}
],
"most_likely_cause": "A cache invalidation event, likely triggered by an upstream configuration change or TTL expiry, caused a thundering herd that overwhelmed the fraud API and degraded checkout.",
"recommended_fix": "Temporarily enable circuit breaking for the fraud API to fail open, scale fraud API replicas, and investigate cache TTL or configuration changes at 14:00 UTC.",
"confidence_score": 8
}
=== Peer Review ===
{
"gaps": [
"The RCA does not verify whether the fraud API timeout is a cause or a symptom.",
"No network-level metrics between the gateway and fraud API were considered."
],
"missed_hypotheses": [
"A targeted DDoS on checkout endpoints could explain selective impact.",
"A certificate expiry on the fraud API could cause sudden timeouts without code changes."
],
"overall_assessment": "The logic is sound but relies heavily on correlational evidence. Network traces or fraud API status page data would strengthen confidence."
}Next steps
Wire this agent into your incident management webhook so it drafts an RCA within seconds of a page. If you need heavier deductive depth for regulatory post-mortems, swap the reasoning model to deepseek-r1-671b. You can explore Oxlo.ai's request-based pricing for long-context agent workloads at https://oxlo.ai/pricing.
Top comments (0)