shashank ms

Posted on Aug 8

Kimi K2.5 Model Review and Comparison

#learnai #oxlo #ai

What we are building

I built a small harness to compare Kimi K2.5 against Llama 3.3 70B and DeepSeek V3.2 on a long-document summarization task. The script generates a synthetic incident report, feeds it to each model on Oxlo.ai, and uses a judge model to score the outputs. If you are selecting a model for research or agentic workflows, this gives you a reproducible starting point.

What you'll need

Python 3.10 or newer
pip install openai
An Oxlo.ai API key from https://portal.oxlo.ai

Step 1: Set up the Oxlo.ai client

We instantiate the OpenAI client pointing at Oxlo.ai. No extra configuration is needed because the platform is fully OpenAI-compatible.

from openai import OpenAI
import os

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key=os.getenv("OXLO_API_KEY", "YOUR_OXLO_API_KEY")
)

Step 2: Create a long-context test document

To test long-context handling, we synthesize a large incident report in memory. This keeps the benchmark self-contained and avoids external dependencies.

base_sections = [
    "Architecture: The payment gateway relies on a three-node PostgreSQL cluster with Patroni for leader election.",
    "Failure mode: At 14:03 UTC, the leader experienced a checkpoint spike, causing replication lag to exceed 30 seconds.",
    "Detection: PagerDuty alerted on 'replication_lag_seconds > 20' at 14:05 UTC.",
    "Mitigation: The on-call engineer failed over to a synchronous replica at 14:12 UTC.",
    "Post-incident: A missing index on the transactions table was identified as the root cause.",
]

long_doc = "\n\n".join(
    f"Incident report segment {i}:\n{section}"
    for i, section in enumerate(base_sections * 200, 1)
)

user_message = (
    "You are given a lengthy incident post-mortem. "
    "Extract the root cause, impact, timeline, and action items. "
    "Respond in a structured format.\n\n" + long_doc
)

Step 3: Define the research agent prompt

We pin the system prompt so every candidate model receives identical instructions. Fair comparison depends on controlling the prompt.

SYSTEM_PROMPT = """You are a senior site reliability engineer.
Read the incident report provided by the user.
Respond with exactly these four sections:
- Root Cause
- Impact
- Timeline
- Action Items
Be concise and accurate. Do not add commentary outside the four sections."""

Step 4: Query Kimi K2.5 and two baselines

Now we loop over the three models. I chose Kimi K2.5 for its chain-of-thought reasoning, Llama 3.3 70B as a general-purpose baseline, and DeepSeek V3.2 for reasoning and coding tasks. Because Oxlo.ai uses flat per-request pricing, running this loop on a massive input does not scale in cost with token count the way token-based providers do. See https://oxlo.ai/pricing for current plan details.

models = {
    "kimi-k2.5": "kimi-k2.5",
    "llama-3.3-70b": "llama-3.3-70b",
    "deepseek-v3.2": "deepseek-v3.2",
}

outputs = {}

for name, model_id in models.items():
    print(f"Running {name}...")
    response = client.chat.completions.create(
        model=model_id,
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": user_message},
        ],
    )
    outputs[name] = response.choices[0].message.content
    print(f"  Received {len(outputs[name])} chars\n")

Step 5: Judge outputs with structured scoring

Reading three summaries side by side is tedious, so we automate evaluation. We reuse Kimi K2.5 as a judge and request JSON output for easy parsing. Oxlo.ai supports JSON mode on compatible models.

import json

judge_prompt = f"""You are an impartial evaluator.
Below are three summaries of the same long incident report.
Score each on accuracy, completeness, and brevity from 1 to 10.
Return strictly JSON with this shape:
{{"kimi-k2.5": {{"accuracy": 0, "completeness": 0, "brevity": 0, "notes": ""}},
  "llama-3.3-70b": {{"accuracy": 0, "completeness": 0, "brevity": 0, "notes": ""}},
  "deepseek-v3.2": {{"accuracy": 0, "completeness": 0, "brevity": 0, "notes": ""}}}}

--- kimi-k2.5 ---
{outputs["kimi-k2.5"]}

--- llama-3.3-70b ---
{outputs["llama-3.3-70b"]}

--- deepseek-v3.2 ---
{outputs["deepseek-v3.2"]}
"""

judge_response = client.chat.completions.create(
    model="kimi-k2.5",
    messages=[
        {"role": "system", "content": "Return only valid JSON. No markdown fences."},
        {"role": "user", "content": judge_prompt},
    ],
    response_format={"type": "json_object"},
)

scores = json.loads(judge_response.choices[0].message.content)
print(json.dumps(scores, indent=2))

Run it

Save the full script to compare.py, export your key, and run it. The script prints character counts as it goes, then emits the judge scores. The numbers below are from one example run and will differ each time.

$ export OXLO_API_KEY="sk-..."
$ python compare.py
Running kimi-k2.5...
  Received 847 chars

Running llama-3.3-70b...
  Received 623 chars

Running deepseek-v3.2...
  Received 912 chars

{
  "kimi-k2.5": {
    "accuracy": 9,
    "completeness": 9,
    "brevity": 8,
    "notes": "Correctly identified the missing index and provided a clean timeline."
  },
  "llama-3.3-70b": {
    "accuracy": 8,
    "completeness": 7,
    "brevity": 9,
    "notes": "Missed the synchronous replica failover detail but stayed very concise."
  },
  "deepseek-v3.2": {
    "accuracy": 9,
    "completeness": 10,
    "brevity": 6,
    "notes": "Comprehensive, but included extra commentary outside the requested sections."
  }
}

Next steps

Swap in Qwen 3 32B or GLM 5 to grow the comparison matrix, or upgrade to Kimi K2.6 for vision and agentic coding tests. Because Oxlo.ai pricing is per request, adding more models does not increase the marginal cost of each long-context call. If you want to productionize this, wrap the loop in a FastAPI endpoint and write results to SQLite for regression tracking.

DEV Community