Monitoring Deep Reasoning Performance: A Step-by-Step Guide

I built a small reasoning monitor to catch bad chains-of-thought before they reach users. It sends hard problems to a deep reasoning model on Oxlo.ai, then scores the resulting trace for coherence, correctness, and hallucination risk. Because Oxlo.ai uses flat per-request pricing, running a second judge model over a long reasoning trace costs the same whether the trace is fifty tokens or five thousand, which makes continuous monitoring practical.

What you'll need

You will need Python 3.10 or newer, the OpenAI SDK, and an Oxlo.ai API key from https://portal.oxlo.ai. Install the SDK with pip.

pip install openai

Step 1: Configure the Oxlo.ai client

First, import the OpenAI SDK and point it at Oxlo.ai. I keep my key in an environment variable.

import os
from openai import OpenAI

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key=os.getenv("OXLO_API_KEY")
)

Step 2: Capture a reasoning trace

I use DeepSeek V3.2 on Oxlo.ai for the reasoning step because it handles coding and logic well, and it sits on the free tier. Here is the system prompt I use to force explicit step numbering.

REASONING_SYSTEM_PROMPT = "Think step by step. Number each step. If you are uncertain, say so explicitly."

The function sends the problem to the model and returns the raw trace.

def get_reasoning_trace(problem: str) -> str:
    response = client.chat.completions.create(
        model="deepseek-v3.2",
        messages=[
            {"role": "system", "content": REASONING_SYSTEM_PROMPT},
            {"role": "user", "content": problem},
        ],
    )
    return response.choices[0].message.content

Step 3: Define the reasoning judge

Now we need a second model to evaluate the trace. I use Llama 3.3 70B as a judge because it follows structured instructions reliably. After a few iterations, this is the prompt that gives consistent JSON output.

REASONING_JUDGE_PROMPT = """You are a reasoning evaluator. Review the user's chain-of-thought and return a JSON object with exactly these keys:
- overall_score: integer 1 to 10
- logical_gaps: list of strings describing any missing or broken steps
- hallucination_risk: string, one of low, medium, high
- summary: string, max 20 words

Be strict. If a step assumes facts not stated or derived, flag it."""

The judge function sends the trace and the original problem together so the judge has full context.

import json

def score_trace(problem: str, trace: str) -> dict:
    user_content = f"Problem:\n{problem}\n\nTrace:\n{trace}"

    response = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=[
            {"role": "system", "content": REASONING_JUDGE_PROMPT},
            {"role": "user", "content": user_content},
        ],
        response_format={"type": "json_object"},
    )
    raw = response.choices[0].message.content
    return json.loads(raw)

Step 4: Log results locally

I append every evaluation to a JSONL file so I can trend scores over time. This keeps the monitor stateless and easy to inspect with standard Unix tools.

from datetime import datetime, timezone

def log_evaluation(problem: str, trace: str, score: dict):
    record = {
        "timestamp": datetime.now(timezone.utc).isoformat(),
        "problem": problem,
        "trace": trace,
        "evaluation": score,
    }
    with open("reasoning_log.jsonl", "a") as f:
        f.write(json.dumps(record) + "\n")

Step 5: Batch monitor and report

Finally, I run a batch of test problems, collect scores, and print a summary. This is the entry point I run in CI every night.

TEST_PROBLEMS = [
    "Write a Python function that returns the n-th Fibonacci number in O(log n) time.",
    "If a train travels 60 km in 40 minutes, how far will it travel in 2.5 hours at the same speed?",
    "Explain why a binary search tree degrades to O(n) lookup and how to fix it.",
]

def run_monitor():
    for problem in TEST_PROBLEMS:
        trace = get_reasoning_trace(problem)
        score = score_trace(problem, trace)
        log_evaluation(problem, trace, score)
        print(f"Score: {score['overall_score']}/10 | Risk: {score['hallucination_risk']} | {score['summary']}")

if __name__ == "__main__":
    run_monitor()

Run it

Export your key and run the script.

export OXLO_API_KEY="sk-oxlo.ai-..."
python monitor.py

You should see output similar to this.

Score: 9/10 | Risk: low | Correct matrix exponentiation approach
Score: 8/10 | Risk: low | Correct proportion but minor unit confusion
Score: 9/10 | Risk: low | Accurate skew explanation with AVL fix

After the run, reasoning_log.jsonl contains one JSON object per line. You can grep it, stream it to Grafana, or load it into a notebook to plot score trends.

Wrap-up

A concrete next step is to wrap the run_monitor loop in a FastAPI endpoint so your inference pipeline can call it asynchronously after each heavy reasoning task. Another is to swap the judge model to Kimi K2.6 on Oxlo.ai when you start evaluating vision-reasoning traces, since it handles multimodal agentic coding well.