Introduction to LLM Model Compression

#engineering #oxlo #ai

We are building a model compression evaluator that routes a single task through a large baseline model and several efficient alternatives on Oxlo.ai, then scores the outputs to find the smallest model that still meets quality requirements. This helps teams cut inference costs without guessing which architecture fits their workload. Because Oxlo.ai uses request-based pricing, running multiple candidates against a long prompt does not multiply your bill the way token-based billing would.

What you'll need

Python 3.10 or newer
An Oxlo.ai API key from https://portal.oxlo.ai
The OpenAI SDK (pip install openai)

Step 1: Configure the client and model slate

I start by initializing the OpenAI-compatible client against Oxlo.ai and defining the baseline model plus the efficient candidates we want to test.

import os
from openai import OpenAI

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key=os.environ["OXLO_API_KEY"]
)

BASELINE_MODEL = "llama-3.3-70b"
CANDIDATES = [
    "deepseek-v3.2",
    "qwen-3-32b",
    "deepseek-v4-flash",
]

Step 2: Build the inference helper

Next I add a small helper that sends a prompt to any Oxlo.ai model and returns the generated text. Keeping this in one function makes the rest of the script easier to read.

def run_model(model_id: str, prompt: str) -> str:
    response = client.chat.completions.create(
        model=model_id,
        messages=[
            {"role": "system", "content": "You are a helpful assistant. Answer concisely."},
            {"role": "user", "content": prompt},
        ],
        temperature=0.2,
    )
    return response.choices[0].message.content

Step 3: Build the LLM judge

To avoid manual review, I use a strong reasoning model on Oxlo.ai to compare each candidate output against the baseline. Here is the system prompt I give the judge.

SYSTEM_PROMPT = """You are an expert evaluator of LLM outputs. Compare the candidate response to the baseline response. Rate the candidate on a scale of 1 to 5, where 5 means the candidate is fully equivalent in quality and 1 means it is unusable. Respond with only a JSON object containing 'score' and 'reason'."""

And here is the judge function that calls Kimi K2.6.

import json

def judge_output(baseline: str, candidate: str, candidate_id: str, task: str) -> dict:
    evaluation_prompt = f"""Task: {task}

Baseline output:
{baseline}

Candidate output ({candidate_id}):
{candidate}

Rate the candidate on a scale of 1 to 5, where 5 means it fully matches baseline quality and 1 means it is unusable. Respond with only a JSON object containing 'score' and 'reason'."""

    response = client.chat.completions.create(
        model="kimi-k2.6",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": evaluation_prompt},
        ],
        temperature=0.1,
    )

    raw = response.choices[0].message.content.strip()
    if raw.startswith("

```"):
        raw = raw.split("```

")[1].replace("json", "").strip()
    return json.loads(raw)

Step 4: Orchestrate the evaluation

Now I wire everything together. The script runs the baseline, runs every candidate, judges the results, and recommends the first candidate that scores at least a 4 out of 5.

def evaluate_compression(prompt: str):
    print(f"Running baseline: {BASELINE_MODEL}")
    baseline_output = run_model(BASELINE_MODEL, prompt)

    results = []
    for model_id in CANDIDATES:
        print(f"Running candidate: {model_id}")
        candidate_output = run_model(model_id, prompt)
        judgement = judge_output(baseline_output, candidate_output, model_id, prompt)
        results.append((model_id, judgement))

    print("\n--- Compression Report ---")
    print(f"Baseline ({BASELINE_MODEL}):\n{baseline_output}\n")

    recommendation = None
    for model_id, judgement in results:
        score = judgement.get("score", 0)
        reason = judgement.get("reason", "No reason provided")
        print(f"{model_id} (score {score}/5): {reason}")
        if score >= 4 and recommendation is None:
            recommendation = model_id

    if recommendation:
        print(f"\nRecommended compressed model: {recommendation}")
    else:
        print("\nNo candidate met the quality bar. Stick with the baseline.")

Run it

Save the script as compress_eval.py, set your API key, and run it with a coding task.

export OXLO_API_KEY="YOUR_OXLO_API_KEY"
python compress_eval.py

Here is what the output looks like when I test a Fibonacci memoization prompt.

Running baseline: llama-3.3-70b
Running candidate: deepseek-v3.2
Running candidate: qwen-3-32b
Running candidate: deepseek-v4-flash

--- Compression Report ---
Baseline (llama-3.3-70b):


```python
def fibonacci(n: int, memo: dict = None) -> int:
    """Return the nth Fibonacci number using memoization."""
    if memo is None:
        memo = {}
    if n in memo:
        return memo[n]
    if n < 2:
        return n
    memo[n] = fibonacci(n - 1, memo) + fibonacci(n - 2, memo)
    return memo[n]
```



deepseek-v3.2 (score 5/5): Functionally identical to baseline with correct type hints and docstring.
qwen-3-32b (score 5/5): Equivalent implementation and documentation quality.
deepseek-v4-flash (score 4/5): Correct logic and type hints, but the docstring omits the memoization detail.

Recommended compressed model: deepseek-v3.2

Wrap-up

You can extend this evaluator by adding latency tracking with time.perf_counter so you report tokens per second or wall-clock savings alongside quality scores. Another useful next step is to turn the script into a FastAPI middleware that dynamically routes incoming requests to the cheapest adequate model on Oxlo.ai based on a cached evaluation for each endpoint.