Unlocking the Power of Chain-of-Thought Reasoning in LLM Models

#learnai #oxlo #ai

We are going to build a support ticket triage agent that thinks out loud before it decides. By forcing chain-of-thought reasoning in the system prompt, we get more reliable category tags and priority scores than a model that jumps straight to an answer. Because Oxlo.ai uses flat per-request pricing, long reasoning traces do not inflate cost, which makes this approach practical even for high-volume queues.

What you'll need

Python 3.10 or newer
An Oxlo.ai API key from https://portal.oxlo.ai
The OpenAI SDK: pip install openai

Step 1: Configure the Oxlo.ai client

Oxlo.ai exposes an OpenAI-compatible endpoint, so we can use the official SDK with a single line pointing the base URL at Oxlo.ai.

from openai import OpenAI

client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")

Step 2: Design the chain-of-thought system prompt

We want the model to write its reasoning inside <thinking> tags and only then output a JSON verdict. This keeps the logic inspectable and easy to separate from the final decision.

SYSTEM_PROMPT = """You are a support ticket triage agent.
Your job is to analyze an incoming customer message and decide:
1. The category (billing, technical, account, or general).
2. The priority (low, medium, high, or critical).
3. A one-sentence summary.
4. A confidence score from 0.0 to 1.0.

First, think step by step inside <thinking> tags. Consider:
- Is the user unable to use the product?
- Are money or data loss involved?
- Is the tone urgent or threatening churn?

After you finish reasoning, output ONLY a JSON object in this exact format:
{
  "category": "...",
  "priority": "...",
  "summary": "...",
  "confidence": 0.0
}
Do not write any text after the JSON."""

Step 3: Build the triage function

Now we call Oxlo.ai with the ticket text. I use kimi-k2.6 because its advanced reasoning capabilities handle the structured chain-of-thought format reliably.

def triage_ticket(ticket_text: str) -> str:
    response = client.chat.completions.create(
        model="kimi-k2.6",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": ticket_text},
        ],
        temperature=0.2,
        max_tokens=1024,
    )
    return response.choices[0].message.content

Step 4: Split reasoning from the final JSON

The raw response contains both the thinking trace and the JSON payload. A small regex pulls them apart so we can log the reasoning and store the structured decision separately.

import json
import re

def parse_response(raw: str) -> dict:
    thinking_match = re.search(r"<thinking>(.*?)</thinking>", raw, re.DOTALL)
    thinking = thinking_match.group(1).strip() if thinking_match else "No reasoning provided"

    # Strip the thinking block and any markdown fences
    json_part = re.sub(r"<thinking>.*?</thinking>", "", raw, flags=re.DOTALL).strip()
    json_part = json_part.removeprefix("

```json").removeprefix("```

").removesuffix("

```

").strip()

    verdict = json.loads(json_part)

    return {
        "thinking": thinking,
        "verdict": verdict,
    }

def run_triage(ticket_text: str) -> dict:
    raw = triage_ticket(ticket_text)
    return parse_response(raw)

Step 5: Wire it into a small CLI

To make this runnable, I added a short script that reads a ticket from a text file and prints the reasoning trace followed by the structured result.

if __name__ == "__main__":
    import sys

    if len(sys.argv) != 2:
        print("Usage: python triage.py ticket.txt")
        sys.exit(1)

    with open(sys.argv[1], "r", encoding="utf-8") as f:
        ticket = f.read()

    result = run_triage(ticket)

    print("=== REASONING ===")
    print(result["thinking"])
    print("\n=== VERDICT ===")
    print(json.dumps(result["verdict"], indent=2))

Run it

Save the following ticket as ticket.txt and run python triage.py ticket.txt.

We were charged $4,200 twice this month and now our API keys stopped working.
This is blocking our entire production deploy. We are considering canceling.
Please fix immediately.

Example output:

=== REASONING ===
The user reports a duplicate charge of $4,200, which is a billing issue.
However, they also state that their API keys stopped working and it is blocking production.
This indicates a technical problem with immediate business impact.
The tone is urgent and mentions churn, so priority should be critical.

=== VERDICT ===
{
  "category": "technical",
  "priority": "critical",
  "summary": "API keys inactive after duplicate billing, blocking production deploy",
  "confidence": 0.95
}

Wrap-up

This agent gives you an auditable reasoning trail for every routing decision. A concrete next step is to wire the output into a Slack webhook so the thinking block posts to a private channel for human review. You could also add a confidence threshold that escalates tickets to a senior agent whenever the score drops below 0.8.

DEV Community