LLM Model Distillation Explained

We are building a lightweight distillation pipeline that turns a large teacher model into a synthetic reasoning dataset, then tests a smaller student model on held-out problems. It helps engineers bootstrap domain-specific datasets without managing surprise token costs.

What you'll need

Python 3.10 or newer, the openai SDK (pip install openai), and an Oxlo.ai API key from https://portal.oxlo.ai. Oxlo.ai flat per-request pricing makes generating hundreds of examples predictable, and you can see plan details at https://oxlo.ai/pricing.

Step 1: Configure the Oxlo.ai client

Import the OpenAI SDK, point it at Oxlo.ai, and verify connectivity with a short call.

from openai import OpenAI

client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")

response = client.chat.completions.create(
    model="deepseek-v3.2",
    messages=[{"role": "user", "content": "Say 'Oxlo.ai client ready'"}],
    max_tokens=20,
)
print(response.choices[0].message.content)

Step 2: Define the teacher system prompt

The teacher needs to emit structured reasoning traces we can reuse. This prompt enforces a strict format so later stages can parse the output.

SYSTEM_PROMPT = (
    "You are a meticulous math and logic teacher. "
    "For every question, first work through the solution step by step inside <thinking> tags. "
    "Then provide the final answer inside <answer> tags. "
    "Be explicit and show all intermediate reasoning."
)

Step 3: Generate a synthetic dataset with the teacher

We loop through seed questions and call the teacher model, kimi-k2.6, to produce reasoning chains. Because Oxlo.ai charges per request rather than per token, long <thinking> blocks do not inflate the cost.

from openai import OpenAI

client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")

questions = [
    "A train travels 120 km in 2 hours. How far will it travel in 5 hours at the same speed?",
    "If a rectangle has a length of 8 cm and a width of 5 cm, what is its area?",
    "What is the sum of the first 10 positive integers?",
]

dataset = []

for q in questions:
    resp = client.chat.completions.create(
        model="kimi-k2.6",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": q},
        ],
    )
    dataset.append({"question": q, "teacher_output": resp.choices[0].message.content})

print(f"Generated {len(dataset)} examples.")

Step 4: Distill reasoning to the student model

We feed the first two teacher examples as few-shot context to a smaller model, qwen-3-32b, and ask it to solve a new problem in the same format. This mimics the first phase of distillation: transferring the teacher's reasoning style.

from openai import OpenAI

client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")

few_shot = ""
for ex in dataset[:2]:
    few_shot += f"Question: {ex['question']}\n{ex['teacher_output']}\n\n"

new_question = "A bakery sells 240 loaves in 4 days. How many loaves will it sell in 10 days at the same rate?"

student_prompt = few_shot + f"Question: {new_question}\n"

student_resp = client.chat.completions.create(
    model="qwen-3-32b",
    messages=[
        {"role": "system", "content": "Follow the reasoning format shown in the examples."},
        {"role": "user", "content": student_prompt},
    ],
)

student_answer = student_resp.choices[0].message.content
print(student_answer)

Step 5: Evaluate the student against the teacher

We generate a fresh teacher answer for the held-out question, then use llama-3.3-70b as a judge to score the student output on a 1-10 scale.

from openai import OpenAI

client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")

teacher_resp = client.chat.completions.create(
    model="kimi-k2.6",
    messages=[
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user", "content": new_question},
    ],
)
teacher_answer = teacher_resp.choices[0].message.content

judge_prompt = (
    "You are a grading assistant. Compare the student answer to the teacher answer.\n\n"
    f"Teacher Answer:\n{teacher_answer}\n\n"
    f"Student Answer:\n{student_answer}\n\n"
    "Respond with a JSON object containing 'score' (1-10) and 'notes'."
)

judge_resp = client.chat.completions.create(
    model="llama-3.3-70b",
    messages=[{"role": "user", "content": judge_prompt}],
    response_format={"type": "json_object"},
)

print(judge_resp.choices[0].message.content)

Run it

Save the snippets in order into distill.py, replace YOUR_OXLO_API_KEY, and run python distill.py. You should see the dataset size printed, the student's <thinking> block, and a JSON grade similar to this:

Generated 3 examples.
<thinking>
The bakery sells 240 loaves in 4 days, so the daily rate is 240 / 4 = 60 loaves per day.
Over 10 days, the total is 60 * 10 = 600 loaves.
</thinking>
<answer>600 loaves</answer>

{"score": 10, "notes": "Student used the same step-by-step structure and arrived at the correct answer."}

Wrap-up

You now have a working distillation agent that produces structured training data and evaluates a student model. Two concrete next steps: scale the question list to a few hundred items and export the dataset to JSONL for fine-tuning with a local framework like Axolotl, or add asyncio concurrency to saturate your Oxlo.ai request quota and generate data faster.