DEV Community

shashank ms
shashank ms

Posted on

Troubleshooting LLM Issues: A Comprehensive Guide

#engineering #oxlo #ai

We are building an LLM Troubleshooter Agent that ingests production error traces, prompt snapshots, and model configs, then returns a structured diagnosis with a concrete fix. It is for engineering teams that run LLMs in production and need to cut incident response time from hours to minutes. Because Oxlo.ai charges a flat rate per request rather than per token, you can paste full stack traces and long prompt histories into the diagnostic context without cost surprises.

What you'll need

Step 1: Verify connectivity

First, I initialize the Oxlo.ai client and send a smoke test to confirm the endpoint and key are live. I use llama-3.3-70b because it starts instantly with no cold start.

from openai import OpenAI

client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")

response = client.chat.completions.create(
    model="llama-3.3-70b",
    messages=[{"role": "user", "content": "ping"}],
    max_tokens=5
)
print("Oxlo.ai status:", response.choices[0].message.content)

Step 2: Define the system prompt

The system prompt locks the model into a strict diagnostic role and forces JSON output. This keeps reasoning consistent across every incident.

SYSTEM_PROMPT = """You are an LLM incident diagnostician. Analyze the provided error context and respond with a JSON object containing:
- root_cause: one sentence describing the failure mode
- severity: "low", "medium", or "critical"
- fix: concrete steps or a corrected prompt snippet
- prevention: a brief guardrail recommendation

Be concise. Do not speculate beyond the evidence."""

Step 3: Build the incident collector

I need a helper that turns scattered logs into a single structured report. This normalizes whatever telemetry we have into one text block.

def build_incident_report(error_text, model_id, prompt_snapshot, temperature=None, max_tokens=None):
    report = f"""[INCIDENT REPORT]
Model: {model_id}
Temperature: {temperature}
Max tokens: {max_tokens}

[PROMPT SNAPSHOT]
{prompt_snapshot}

[ERROR TRACE]
{error_text}
"""
    return report

Step 4: Run the diagnostic

Now I send the report to the model with JSON mode enabled so the response parses cleanly into a Python dict. I keep temperature low to reduce hallucination.

import json

def diagnose(report):
    response = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": report},
        ],
        response_format={"type": "json_object"},
        temperature=0.2,
    )
    return json.loads(response.choices[0].message.content)

Step 5: Generate the repair

Diagnosis alone is not enough. I run a second pass against deepseek-v3.2 to produce a corrected prompt or code patch based on the root cause. This separates reasoning from generation and usually yields cleaner fixes.

def generate_repair(diagnosis, original_prompt):
    repair_prompt = f"""Given this diagnosis:
{json.dumps(diagnosis, indent=2)}

Produce a corrected version of the original prompt or a Python code patch that prevents the issue. Return only the corrected prompt or code inside a JSON object with key "artifact"."""

    response = client.chat.completions.create(
        model="deepseek-v3.2",
        messages=[
            {"role": "system", "content": "You are a senior ML engineer. Emit only the requested JSON."},
            {"role": "user", "content": repair_prompt},
        ],
        response_format={"type": "json_object"},
        temperature=0.1,
    )
    return json.loads(response.choices[0].message.content)

Run it

Here is a full end-to-end test with a realistic failure: a JSON decode error caused by the model wrapping output in markdown fences. I print both the diagnosis and the repair artifact.

if __name__ == "__main__":
    incident = build_incident_report(
        error_text='json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)',
        model_id="llama-3.3-70b",
        prompt_snapshot="You are a data extractor. Return JSON.\n\nExtract name and age from: Alice, 30",
        temperature=0.7,
        max_tokens=256
    )

    diagnosis = diagnose(incident)
    print("Diagnosis:", json.dumps(diagnosis, indent=2))

    repair = generate_repair(diagnosis, incident)
    print("Repair:", json.dumps(repair, indent=2))

Example output:

Diagnosis: {
  "root_cause": "The model returned markdown fenced JSON instead of raw JSON because the prompt did not explicitly forbid explanatory text.",
  "severity": "medium",
  "fix": "Append 'Return only raw JSON. No markdown, no explanations.' to the system prompt.",
  "prevention": "Add a JSON schema constraint and validate output with json.loads before downstream parsing."
}
Repair: {
  "artifact": "You are a data extractor. Return only raw JSON. No markdown, no explanations.\n\nExtract name and age from: Alice, 30\n\nExample output format:\n{\"name\": \"...\", \"age\": ...}"
}

Wrap-up

From here, you can wire the agent into your CI pipeline to auto-diagnose staging failures, or swap in kimi-k2.6 when you need vision support for dashboard screenshots. If you want to scale further, Oxlo.ai's request-based pricing means you can batch process hundreds of long trace logs without token meter anxiety.

Top comments (0)