LLM Explainability Techniques

#learnai #oxlo #ai

We're going to build an LLM Decision Explainer that wraps any prompt with chain-of-thought extraction and contrastive attribution so you can see why a model generated a specific answer. It is a lightweight debugging tool for developers who need to audit reasoning traces without managing infrastructure. We will run everything against Oxlo.ai's request-based API so cost stays flat even when we send long context windows for analysis.

What you'll need

Python 3.10 or higher
An Oxlo.ai API key from https://portal.oxlo.ai
The OpenAI SDK: pip install openai

Step 1: Capture a baseline response

We start by sending a user question to Oxlo.ai and recording the raw answer. This gives us the decision we need to explain.

from openai import OpenAI

client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")

user_question = "Should I use a monorepo for a microservices project with six teams?"

response = client.chat.completions.create(
    model="llama-3.3-70b",
    messages=[
        {"role": "user", "content": user_question},
    ],
)

baseline_answer = response.choices[0].message.content
print("BASELINE ANSWER:\n", baseline_answer)

Step 2: Extract a chain-of-thought trace

Next, we re-run the same question with a system prompt that forces the model to emit its reasoning before its conclusion. This is the simplest form of interpretability at the prompt layer.

from openai import OpenAI

client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")

user_question = "Should I use a monorepo for a microservices project with six teams?"

SYSTEM_PROMPT_COT = """You are a helpful assistant. Before you give your final answer, think step by step inside <thinking> tags. Consider trade-offs, constraints, and alternatives. After you close the </thinking> tag, provide your final answer."""

cot_response = client.chat.completions.create(
    model="llama-3.3-70b",
    messages=[
        {"role": "system", "content": SYSTEM_PROMPT_COT},
        {"role": "user", "content": user_question},
    ],
)

cot_output = cot_response.choices[0].message.content
print(cot_output)

Step 3: Generate a contrastive explanation

Real explainability requires knowing what would change the outcome. We ask the model to identify the minimal change to the input that would flip its recommendation, which surfaces hidden assumptions.

from openai import OpenAI

client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")

user_question = "Should I use a monorepo for a microservices project with six teams?"
baseline_answer = "A monorepo can work, but with six teams you may face merge contention and slow CI pipelines. Consider a hybrid or polyrepo approach unless you have strong tooling."

contrastive_prompt = f"""The user asked: "{user_question}"
Your baseline answer was: "{baseline_answer}"

Now, perform a contrastive analysis. Describe the smallest realistic change to the user's situation, constraints, or question wording that would cause you to recommend the opposite approach. Be specific."""

contrastive_response = client.chat.completions.create(
    model="llama-3.3-70b",
    messages=[
        {"role": "system", "content": "You are an analytical assistant that identifies decision boundaries."},
        {"role": "user", "content": contrastive_prompt},
    ],
)

print("CONTRASTIVE EXPLANATION:\n", contrastive_response.choices[0].message.content)

Step 4: Add input attribution via span highlighting

We now ask the model to quote the exact words or phrases from the original prompt that most influenced its answer. This simulates attribution without needing token-level gradients.

from openai import OpenAI

client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")

user_question = "Should I use a monorepo for a microservices project with six teams?"
baseline_answer = "A monorepo can work, but with six teams you may face merge contention and slow CI pipelines. Consider a hybrid or polyrepo approach unless you have strong tooling."

attribution_prompt = f"""User question: "{user_question}"
Your answer: "{baseline_answer}"

List the specific words or phrases from the user's question that most influenced your answer. For each phrase, give a one-sentence justification of its weight in your decision."""

attribution_response = client.chat.completions.create(
    model="llama-3.3-70b",
    messages=[
        {"role": "system", "content": "You are a precise attribution engine. Only reference text that actually appears in the user question."},
        {"role": "user", "content": attribution_prompt},
    ],
)

print("ATTRIBUTION:\n", attribution_response.choices[0].message.content)

Step 5: Package the explainer into a reusable agent

Finally, we wrap the three techniques into a single Python class with a unified system prompt so we can call it like an agent. First, define the system prompt the agent uses.

EXPLAINER_SYSTEM_PROMPT = """You are an LLM Decision Explainer. Your job is to help developers audit model outputs.

When given a user question and a baseline answer, produce three sections:
1. Chain-of-Thought Trace: Reconstruct the reasoning steps that likely led to the answer.
2. Contrastive Boundary: Identify the smallest change to the input that would flip the conclusion.
3. Input Attribution: Quote the exact words or phrases from the user question that most influenced the answer, with a brief justification for each.

Be concise, technical, and grounded in the provided text. Do not invent facts not present in the question or answer."""

Now the class implementation.

from openai import OpenAI

class OxloExplainer:
    def __init__(self, api_key: str, model: str = "llama-3.3-70b"):
        self.client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key=api_key)
        self.model = model

    def explain(self, user_question: str, baseline_answer: str) -> str:
        payload = f"""User Question:
{user_question}

Baseline Answer:
{baseline_answer}

Provide the three explainability sections requested in your instructions."""

        response = self.client.chat.completions.create(
            model=self.model,
            messages=[
                {"role": "system", "content": EXPLAINER_SYSTEM_PROMPT},
                {"role": "user", "content": payload},
            ],
        )
        return response.choices[0].message.content

Run it

Here is how to invoke the finished agent and what the output looks like for our monorepo question.

explainer = OxloExplainer(api_key="YOUR_OXLO_API_KEY")
report = explainer.explain(
    user_question="Should I use a monorepo for a microservices project with six teams?",
    baseline_answer="A monorepo can work, but with six teams you may face merge contention and slow CI pipelines. Consider a hybrid or polyrepo approach unless you have strong tooling."
)
print(report)

Example output:

Chain-of-Thought Trace:
The model weighed team scale against tooling overhead. Six teams implies concurrent feature work, which increases blast radius in a single repository. The answer prioritized CI speed and merge safety over code colocation.

Contrastive Boundary:
If the user had stated, "We have a high-performance build system like Bazel and dedicated platform engineers," the recommendation would likely flip to favoring a monorepo because the primary constraint, CI latency, would be mitigated.

Input Attribution:
- "six teams": This quantity triggered the scaling concern. With fewer teams, the answer might have endorsed a monorepo.
- "microservices": Implied service independence, making polyrepo logically consistent and reducing the coupling argument for monorepo.

Next steps

Wire the explainer into your eval pipeline so every failed test case automatically generates an attribution report. You can also swap in a reasoning model like deepseek-v3.2 or qwen-3-32b on Oxlo.ai for more nuanced chain-of-thought traces on complex prompts.

DEV Community