LLM Model Interpretability for Decision Making: Best Practices and Techniques

#learnai #oxlo #ai

We are going to build a Decision Audit Agent that generates a business recommendation and then interrogates its own reasoning trace for hidden assumptions and overconfidence. This gives you a practical interpretability layer for high-stakes LLM workflows without needing a separate explainability pipeline. You can run the whole thing on Oxlo.ai using standard OpenAI SDK calls.

What you'll need

Python 3.10 or newer
The OpenAI SDK: pip install openai
An Oxlo.ai API key from https://portal.oxlo.ai

Step 1: Set up the Oxlo.ai client

First we initialize the OpenAI-compatible client pointing at Oxlo.ai so every subsequent call routes through the same endpoint.

import json
from openai import OpenAI

client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")

Step 2: Define the decision agent and its system prompt

We use DeepSeek V3.2 on Oxlo.ai for the initial reasoning step because it handles structured coding and reasoning tasks reliably. The system prompt forces the model to expose its chain of thought before returning a verdict.

SYSTEM_PROMPT = """You are a senior business analyst. Evaluate the capital allocation scenario and return a structured JSON decision.

Follow this exact schema:
{
  "scenario_summary": "string",
  "chain_of_thought": "string (your step-by-step reasoning)",
  "decision": "approve | reject | defer",
  "confidence_score": number between 0 and 1,
  "key_factors": ["string"]
}

Rules:
- Write the chain_of_thought first, then derive the decision.
- Be explicit about assumptions.
- Calibrate confidence honestly. Do not default to 0.9."""

def generate_decision(scenario: str) -> dict:
    response = client.chat.completions.create(
        model="deepseek-v3.2",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": scenario},
        ],
        response_format={"type": "json_object"},
        temperature=0.2,
    )
    return json.loads(response.choices[0].message.content)

Step 3: Build the interpretability auditor

Now we feed the reasoning trace into Llama 3.3 70B to audit for cognitive biases, missing evidence, and confidence calibration. This second pass is the interpretability layer.

AUDIT_PROMPT = """You are an interpretability auditor. Analyze the chain-of-thought reasoning trace from another AI analyst.

Return a JSON object with this schema:
{
  "biases_detected": ["string"],
  "unstated_assumptions": ["string"],
  "confidence_calibration": "string (overconfident | underconfident | well-calibrated)",
  "missing_evidence": ["string"],
  "audit_verdict": "string"
}

Be specific. Quote problematic phrases from the reasoning trace when possible."""

def audit_reasoning(decision: dict) -> dict:
    reasoning = decision["chain_of_thought"]
    response = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=[
            {"role": "system", "content": AUDIT_PROMPT},
            {"role": "user", "content": f"Reasoning trace to audit:\n{reasoning}"},
        ],
        response_format={"type": "json_object"},
        temperature=0.1,
    )
    return json.loads(response.choices[0].message.content)

Step 4: Wire the pipeline together

We connect both stages so the scenario flows from decision generation into the audit. The final script prints the structured decision and the interpretability report.

def main():
    scenario = (
        "We are considering a $2M investment in a new geographic market. "
        "The market is growing 15% YoY, but we have no local partnerships. "
        "Competitors entered last year and are gaining share. "
        "Our internal data suggests we can capture 5% market share within 18 months."
    )

    print("=== GENERATING DECISION ===")
    decision = generate_decision(scenario)
    print(json.dumps(decision, indent=2))

    print("\n=== AUDITING REASONING ===")
    audit = audit_reasoning(decision)
    print(json.dumps(audit, indent=2))

if __name__ == "__main__":
    main()

Run it

Save the complete script as decision_audit.py and run it. You should see a structured decision followed by an audit that flags weak points in the reasoning.

$ python decision_audit.py
=== GENERATING DECISION ===
{
  "scenario_summary": "Evaluate $2M investment in new geographic market with 15% YoY growth, no local partnerships, and competitor presence.",
  "chain_of_thought": "The 15% YoY growth is attractive, but the lack of local partnerships is a major execution risk. Competitors have a one-year head start, which means customer acquisition costs will be higher than internal projections assume. The 5% market share target is plausible only if we secure at least one local distributor within six months. Without that, the downside exposure exceeds our risk threshold. I am hedging because the plan lacks contingency details.",
  "decision": "defer",
  "confidence_score": 0.65,
  "key_factors": [
    "15% YoY market growth",
    "absence of local partnerships",
    "competitor head start",
    "5% market share target",
    "execution risk on distribution"
  ]
}

=== AUDITING REASONING ===
{
  "biases_detected": [
    "Anchoring on the 5% market share figure without questioning the underlying data source."
  ],
  "unstated_assumptions": [
    "Assumes customer acquisition costs scale linearly with competitor presence.",
    "Assumes a local distributor can be secured within six months."
  ],
  "confidence_calibration": "well-calibrated",
  "missing_evidence": [
    "No mention of regulatory barriers in the new geography.",
    "No analysis of currency or repatriation risk for the $2M capital."
  ],
  "audit_verdict": "The reasoning is directionally sound but relies on two critical unstated assumptions. Recommend requiring a partnership term sheet and regulatory review before revisiting the decision."
}

Wrap-up

You now have a working two-stage interpretability pipeline. A concrete next step is to wrap this in a FastAPI endpoint so every production recommendation gets logged with its audit trail. You could also swap in kimi-k2.6 on Oxlo.ai if you want the agent to analyze financial charts or vision inputs as part of the decision context.

DEV Community