Introduction to LLM Interpretability

#learnai #oxlo #ai

Most interpretability research assumes you own the weights, but production engineers usually only have an API. In this tutorial we will build a black-box prompt ablation tool that identifies which sentences in a prompt actually drive the model's output. It is useful for debugging agents, auditing system prompts, or understanding third-party model behavior.

What you'll need

Python 3.10+
The OpenAI SDK: pip install openai
An Oxlo.ai API key from https://portal.oxlo.ai

Step 1: Scaffold the project and target prompt

I start by initializing the Oxlo.ai client. Because Oxlo.ai is fully OpenAI-compatible, the only difference is the base_url.

from openai import OpenAI

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key="YOUR_OXLO_API_KEY"
)

TARGET_PROMPT = """You are a travel assistant. The user is in a hurry and has a premium account.
User: I need a flight from NYC to London today. What is the best option?"""

Step 2: Generate sentence-level ablations

To isolate influence, I create masked variants of the prompt where one sentence at a time is replaced with [...]. This is a classic ablation technique adapted for API-only access.

import re

def ablate_sentences(text: str):
    sentences = re.split(r'(?<=[.!?])\s+', text.strip())
    variants = []
    for i in range(len(sentences)):
        masked = sentences.copy()
        masked[i] = "[...]"
        variants.append(" ".join(masked))
    return variants

ablations = ablate_sentences(TARGET_PROMPT)
print(f"Generated {len(ablations)} ablated variants")

Step 3: Run the ablation batch against Oxlo.ai

I keep temperature at 0.0 so differences between runs come from the ablation, not sampling noise. I use deepseek-v3.2 because it is fast and handles reasoning well, and because Oxlo.ai's flat per-request pricing means I can run a sweep of twenty long prompts without the cost scaling with token count.

def get_completion(prompt: str) -> str:
    response = client.chat.completions.create(
        model="deepseek-v3.2",
        messages=[{"role": "user", "content": prompt}],
        temperature=0.0,
    )
    return response.choices[0].message.content

baseline = get_completion(TARGET_PROMPT)
ablation_responses = [get_completion(v) for v in ablations]

Step 4: Define the interpreter system prompt

The analyzer is itself an LLM call. I use a strong reasoning model, kimi-k2.6, and lock its behavior with a strict system prompt.

INTERPRETER_SYSTEM_PROMPT = """You are an LLM interpretability analyst.
You will receive a baseline prompt, its baseline response, and a set of ablated prompts with their responses.
For each ablation, state whether masking that sentence caused a meaningful change in content, tone, or structure.
Then summarize which sentences were most influential and explain why.
Be concise. Use bullet points."""

Step 5: Analyze the deltas and emit the report

I package every variant and its output into a single context window and ask the interpreter to find the drivers. Even with long inputs, kimi-k2.6 handles the load, and Oxlo.ai's request-based pricing keeps the cost predictable no matter how much accumulated text I send.

analysis_input = f"""Baseline prompt:
{TARGET_PROMPT}

Baseline response:
{baseline}

Ablations:
"""
for idx, (variant, resp) in enumerate(zip(ablations, ablation_responses)):
    analysis_input += f"\n--- Variant {idx} ---\nPrompt: {variant}\nResponse: {resp}\n"

response = client.chat.completions.create(
    model="kimi-k2.6",
    messages=[
        {"role": "system", "content": INTERPRETER_SYSTEM_PROMPT},
        {"role": "user", "content": analysis_input},
    ],
    temperature=0.2,
)

report = response.choices[0].message.content
print(report)

Run it

Save the complete script as probe.py and run it from your terminal.

python probe.py

Example output:

Generated 2 ablated variants

INTERPRETABILITY REPORT

- Variant 0 (masked "You are a travel assistant."):
  Minimal change. The model still assumes an assistant role, likely due to the user question format.

- Variant 1 (masked "The user is in a hurry and has a premium account."):
  SIGNIFICANT CHANGE. The baseline recommended an express business-class fare, while this variant returned a generic economy search. This sentence is the primary driver of urgency and upsell behavior.

Summary:
The model's output is most sensitive to the explicit persona and status cues. Removing urgency/premium context strips the recommendation of its prioritization logic. I recommend versioning this sentence carefully in production prompts.

Wrap-up and next steps

This pattern works for any prompt you can version in git. A concrete next step is wiring the probe into a CI job that fails when a prompt edit unexpectedly shifts model behavior on a golden set of inputs. For longer documents, swap in deepseek-v4-flash with its 1M context window, knowing that Oxlo.ai's flat per-request pricing keeps the cost predictable no matter how much text you ablate.

DEV Community