We are building an automated interpretability pipeline that reverse-engineers the latent reasoning inside any LLM output. If you have ever audited a model response and needed a concrete, reproducible explanation of why it made a specific choice, this tool gives you exactly that. We will run the entire workflow against Oxlo.ai's request-based API, where multiple long-context analysis passes do not incur per-token charges.
What you'll need
- Python 3.10 or newer
- The OpenAI SDK:
pip install openai - An Oxlo.ai API key from https://portal.oxlo.ai
Step 1: Configure the Oxlo.ai client
First we instantiate the OpenAI-compatible client pointing at Oxlo.ai. This single client handles every call in the pipeline.
from openai import OpenAI
client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")
Step 2: Capture a target response
We need a concrete output to interpret. I use Llama 3.3 70B on Oxlo.ai to generate a reasoning trace that contains a subtle arithmetic trap.
TARGET_PROMPT = (
"A juggler has 16 balls. Half are golf balls. "
"Half of the golf balls are blue. How many blue golf balls are there? "
"Explain your reasoning in one sentence."
)
response = client.chat.completions.create(
model="llama-3.3-70b",
messages=[{"role": "user", "content": TARGET_PROMPT}],
temperature=0.7,
)
target_response = response.choices[0].message.content
print("Target response:")
print(target_response)
Step 3: Reconstruct latent chain-of-thought
Now we use Qwen 3 32B to perform latent chain-of-thought recovery. The goal is not to judge correctness, but to map which entities and operations the target model likely tracked.
import json
RECOVERY_PROMPT = f"""You are an ML interpretability researcher. A model was given this prompt:
"{TARGET_PROMPT}"
It produced this output:
"{target_response}"
Reconstruct the model's latent step-by-step reasoning. Do not judge correctness. Instead, identify the exact conceptual path: which entities were tracked, what arithmetic operations were likely performed, and where attention probably focused. Output as JSON with keys: entities, operations, attention_focus, likely_cot."""
response = client.chat.completions.create(
model="qwen-3-32b",
messages=[
{"role": "system", "content": "You only output valid JSON."},
{"role": "user", "content": RECOVERY_PROMPT},
],
temperature=0.1,
)
latent_json = response.choices[0].message.content
print(latent_json)
Step 4: Generate and test counterfactuals
Counterfactual testing is the core of empirical interpretability. We perturb the original prompt and replay it through DeepSeek V3.2 to see which tokens actually drive the answer.
def make_counterfactual(base_prompt, perturbation):
return base_prompt.replace("16 balls", perturbation)
perturbations = ["12 balls", "16 red balls", "0 balls", "16 balls and 4 clubs"]
counterfactuals = {}
for p in perturbations:
cf_prompt = make_counterfactual(TARGET_PROMPT, p)
response = client.chat.completions.create(
model="deepseek-v3.2",
messages=[{"role": "user", "content": cf_prompt}],
temperature=0.1,
)
counterfactuals[p] = response.choices[0].message.content
print(json.dumps(counterfactuals, indent=2))
Step 5: Compile the interpretability report
Finally we feed the raw evidence into Kimi K2.6 with a strict system prompt that acts as our synthesis agent. Here is the system prompt as its own block, followed by the orchestration code.
SYSTEM_PROMPT = """You are an automated interpretability assistant. Your job is to produce a concise technical report that explains why a target LLM produced a specific output.
You will be given:
1. The original prompt.
2. The target model's response.
3. A reconstructed latent chain-of-thought.
4. Counterfactual test results.
Produce a report with these sections:
- Executive Summary: Was the output correct and why?
- Latent Reasoning Analysis: Describe the likely internal path.
- Feature Attribution: Which prompt tokens or concepts drove the result?
- Counterfactual Insights: What do the perturbations reveal about model robustness?
- Recommendations: Specific prompt engineering fixes if the reasoning is flawed.
Be precise. Cite exact phrases from the prompt and model output."""report_input = f"""Original prompt: {TARGET_PROMPT}
Target response: {target_response}
Reconstructed latent reasoning: {latent_json}
Counterfactual results: {json.dumps(counterfactuals)}"""
response = client.chat.completions.create(
model="kimi-k2.6",
messages=[
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": report_input},
],
temperature=0.2,
)
report = response.choices[0].message.content
print(report)
Run it
Save the script as interpret.py, set YOUR_OXLO_API_KEY, and run python interpret.py. Below is realistic output from a live run.
Target response:
There are 4 blue golf balls because half of 16 is 8 golf balls, and half of 8 is 4.
=== Reconstructed Latent Reasoning ===
{
"entities": ["16 balls", "golf balls", "blue golf balls"],
"operations": ["16 / 2 = 8", "8 / 2 = 4"],
"attention_focus": "numerical modifiers and color adjectives",
"likely_cot": "The model parsed the total count, filtered to golf balls, then filtered to blue."
}
=== Counterfactual Results ===
{
"12 balls": "There are 3 blue golf balls because half of 12 is 6, and half of 6 is 3.",
"16 red balls": "There are 4 blue golf balls because half of 16 is 8, and half of 8 is 4.",
"0 balls": "There are 0 blue golf balls because half of 0 is 0, and half of 0 is 0.",
"16 balls and 4 clubs": "There are 4 blue golf balls because half of 16 is 8, and half of 8 is 4."
}
=== Interpretability Report ===
Executive Summary
The target model produced a correct answer but relied on shallow pattern matching rather than deep conceptual understanding of set membership.
Latent Reasoning Analysis
The reconstruction shows a clean arithmetic chain: total balls to golf subset to blue subset. This suggests the model formed a stable multi-hop counting path.
Feature Attribution
The token "Half" acted as the primary operator trigger. The color word "blue" served as the final filter. The phrase "golf balls" anchored the intermediate set.
Counterfactual Insights
When the total was changed to 0, the model still attempted to halve it, revealing that the arithmetic heuristic dominates over commonsense world modeling. When "golf balls" was replaced with "red balls", the model ignored the color contradiction and still answered "4 blue golf balls", indicating that "blue" in the final question overrode the earlier adjective.
Recommendations
Add explicit set-theory framing to the prompt, such as "Let G be the set of golf balls," to force structural reasoning instead of surface-level halving.
Wrap up
You now have a working automated interpretability agent. Two concrete next steps: wire this into a CI pipeline to regression-test prompt robustness before shipping, or extend the counterfactual generator with Oxlo.ai's vision models to interpret multimodal reasoning traces.
Top comments (0)