Explainability in LLM Models for High-Explainability Tasks

#learnai #oxlo #ai

Support teams cannot act on black-box classifications. In this tutorial we will build a ticket triage agent that outputs a structured decision together with an explicit chain-of-thought reasoning trace. The result is a system you can audit, debug, and ship to production today on Oxlo.ai.

What you'll need

Python 3.10 or newer, the OpenAI SDK, and an Oxlo.ai API key.

pip install openai

Grab an API key from https://portal.oxlo.ai. Oxlo.ai uses flat per-request pricing (see https://oxlo.ai/pricing), so adding long system prompts or few-shot examples for explainability does not inflate your bill the way token-based metering would.

Step 1: Configure the Oxlo.ai client

Oxlo.ai exposes an OpenAI-compatible endpoint. Instantiate the client once with the Oxlo.ai base URL and your key.

from openai import OpenAI
import json
import re

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key="YOUR_OXLO_API_KEY"
)

Step 2: Define the explainability schema and system prompt

We force the model to separate reasoning from the final decision by wrapping the thought process in XML tags before emitting JSON. This makes every citation auditable.

SYSTEM_PROMPT = """You are a support ticket triage agent.
Analyze the ticket and produce:
1. A step-by-step reasoning trace inside <reasoning> tags. Cite exact phrases from the ticket.
2. A JSON object after the reasoning with keys:
   - priority: "low", "medium", or "high"
   - category: "billing", "technical", or "account"
   - sentiment: "negative", "neutral", or "positive"
   - cited_phrases: list of strings
   - reasoning_summary: one sentence

Rules:
- Output the <reasoning> block first.
- Output only the JSON object after the reasoning block, with no markdown fences.
- If the user asks for a refund or threatens churn, mark priority "high"."""

Step 3: Format the incoming ticket

Keep the user message minimal and structured so the model spends its capacity on analysis, not parsing.

def format_ticket(ticket: dict) -> str:
    return f"""Subject: {ticket['subject']}
Body: {ticket['body']}
Customer Tier: {ticket.get('tier', 'standard')}
Date: {ticket.get('date', 'unknown')}"""

Step 4: Query the model and extract the reasoning

We call Oxlo.ai with the system prompt, then use a regex to split the chain-of-thought from the structured JSON decision. I use kimi-k2.6 here because it handles advanced reasoning and agentic tasks well, but you can swap in llama-3.3-70b or qwen-3-32b without changing any other code.

def triage_ticket(ticket: dict, model: str = "kimi-k2.6"):
    user_message = format_ticket(ticket)

    response = client.chat.completions.create(
        model=model,
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": user_message},
        ],
        temperature=0.1,
    )

    raw = response.choices[0].message.content

    reasoning_match = re.search(r"<reasoning>(.*?)</reasoning>", raw, re.DOTALL)
    reasoning = reasoning_match.group(1).strip() if reasoning_match else "No reasoning provided."

    json_match = re.search(r"</reasoning>\s*(\{.*\})", raw, re.DOTALL)
    if not json_match:
        raise ValueError("Could not find JSON decision in model output.")

    decision = json.loads(json_match.group(1))
    decision["raw_reasoning"] = reasoning
    return decision

Step 5: Add a validation guardrail

High-explainability tasks need guardrails. We validate that the reasoning is non-empty and that the decision contains required keys before accepting it.

REQUIRED_KEYS = {"priority", "category", "sentiment", "cited_phrases", "reasoning_summary"}

def validate_decision(decision: dict) -> dict:
    missing = REQUIRED_KEYS - decision.keys()
    if missing:
        raise ValueError(f"Decision missing keys: {missing}")

    if not decision.get("raw_reasoning"):
        raise ValueError("Reasoning trace is empty.")

    if decision["priority"] == "high" and not any(
        "refund" in p.lower() or "cancel" in p.lower() for p in decision.get("cited_phrases", [])
    ):
        print("Warning: high priority without explicit refund or cancel citation.")

    return decision

def explainable_triage(ticket: dict):
    decision = triage_ticket(ticket)
    return validate_decision(decision)

Run it

Call the finished agent with a real support ticket and inspect both the reasoning and the structured output.

ticket = {
    "subject": "Urgent: double charged this month",
    "body": "I was charged twice on my credit card. I need a refund immediately or I will cancel my account. This is unacceptable.",
    "tier": "premium",
    "date": "2025-01-15"
}

result = explainable_triage(ticket)

print("--- Reasoning ---")
print(result["raw_reasoning"])
print("\n--- Structured Decision ---")
print(json.dumps({k: v for k, v in result.items() if k != "raw_reasoning"}, indent=2))

Example output:

--- Reasoning ---
The customer explicitly states "I was charged twice on my credit card," which indicates a billing error. They demand a refund with the phrase "I need a refund immediately," and threaten churn with "I will cancel my account." The phrase "This is unacceptable" signals strong negative sentiment. Given the explicit refund request and churn threat, priority must be high.

--- Structured Decision ---
{
  "priority": "high",
  "category": "billing",
  "sentiment": "negative",
  "cited_phrases": [
    "I was charged twice on my credit card",
    "I need a refund immediately",
    "I will cancel my account"
  ],
  "reasoning_summary": "High priority billing issue with refund request and churn threat."
}

Wrap-up

You now have an auditable triage agent that cites its sources. Two concrete next steps: first, swap in qwen-3-32b or deepseek-v3.2 on Oxlo.ai to compare reasoning depth, which is painless because Oxlo.ai flat per-request pricing does not penalize long-context prompts or multi-turn chains. Second, persist the reasoning traces to an observability store or SQLite table so auditors can search citations later.

DEV Community