shashank ms

Posted on Jul 24

DeepSeek R1 Model Architecture and Benchmarks

#engineering #oxlo #ai

What we're building

I keep a lightweight benchmark harness in my repo to sanity-check model upgrades before they hit production. In this tutorial we will build a reasoning benchmark that runs the same complex algorithmic prompt through DeepSeek R1 671B MoE, Llama 3.3 70B, and Qwen 3 32B on Oxlo.ai, then grades the outputs with Kimi K2.6. Because Oxlo.ai charges per request rather than per token (see pricing), feeding a 2,000-token system prompt to four models costs the same as a one-line greeting.

What you'll need

Python 3.10+
pip install openai
An Oxlo.ai API key from https://portal.oxlo.ai

Step 1: Instantiate the Oxlo.ai client

We point the OpenAI SDK at Oxlo.ai and verify connectivity to the DeepSeek R1 endpoint before we burn any real tasks.

from openai import OpenAI
import os

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key=os.environ.get("OXLO_API_KEY")
)

# Verify DeepSeek R1 is reachable
response = client.chat.completions.create(
    model="deepseek-r1-671b",
    messages=[{"role": "user", "content": "Respond with OK"}],
    max_tokens=10
)
print(response.choices[0].message.content)

Step 2: Define the reasoning task

We need a prompt that stresses the Mixture-of-Experts architecture. I use a constrained optimization problem that requires step-by-step reasoning and a working code solution.

REASONING_PROMPT = """You have N servers with CPU capacities C_i and memory capacities M_i.
You must schedule K containers, each with requirements (c_j, m_j).
Find the minimum number of servers needed to host all containers using a first-fit-decreasing bin-packing heuristic, then prove why this is within 11/9 OPT + 1.
Return your analysis and a Python implementation."""

The system prompt

This system prompt forces the model to expose its chain-of-thought before writing code, which is where DeepSeek R1's deep reasoning architecture typically separates itself from dense models.

SYSTEM_PROMPT = """You are a senior algorithms engineer.
1. Analyze the problem formally. State assumptions and complexity targets.
2. Outline your reasoning step by step.
3. Write clean, commented Python code.
4. Provide a correctness argument.
Be concise but thorough."""

Step 3: Build the benchmark runner

The runner sends the identical message list to each model and stores the raw text. Because Oxlo.ai uses flat per-request pricing, we do not need to truncate the system prompt to save tokens.

MODELS = ["deepseek-r1-671b", "llama-3.3-70b", "qwen-3-32b"]

def run_benchmark(model_id):
    resp = client.chat.completions.create(
        model=model_id,
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": REASONING_PROMPT},
        ],
        temperature=0.2,
        max_tokens=2048
    )
    return resp.choices[0].message.content

results = {}
for m in MODELS:
    print(f"Running {m} ...")
    results[m] = run_benchmark(m)
    print("Done.\n")

Step 4: Grade outputs with a judge model

Instead of reading every response manually, we use Kimi K2.6 to score each answer on reasoning clarity, correctness, and code quality, and ask for strict JSON back.

import json

JUDGE_PROMPT = """You are an expert reviewer. Below are three answers to the same algorithms problem.
Rate each on reasoning (1-10), correctness (1-10), and code quality (1-10).
Return ONLY a JSON object where keys are model names and values are dicts of the three scores.

{answers}"""

formatted = "\n\n---\n\n".join(
    [f"Model: {k}\n{v}" for k, v in results.items()]
)

judge_resp = client.chat.completions.create(
    model="kimi-k2.6",
    messages=[
        {"role": "system", "content": "You output valid JSON only."},
        {"role": "user", "content": JUDGE_PROMPT.format(answers=formatted)}
    ],
    response_format={"type": "json_object"},
    temperature=0.1
)

scores = json.loads(judge_resp.choices[0].message.content)
print(json.dumps(scores, indent=2))

Step 5: Compile the report

We merge the scores and response excerpts into a markdown file we can check into git or attach to a pull request.

report_lines = ["# Reasoning Benchmark Report\n"]
for m in MODELS:
    report_lines.append(f"## {m}")
    report_lines.append(f"Scores: {scores.get(m, {})}")
    report_lines.append(f"Excerpt: {results[m][:300]}...\n")

with open("report.md", "w") as f:
    f.write("\n".join(report_lines))

print("Report written to report.md")

Run it

Save the full script as benchmark.py, export your key, and execute it.

export OXLO_API_KEY="sk-..."
python benchmark.py

You should see OK from the connectivity check, followed by progress lines as each model responds. The terminal then prints a JSON object with the judge's numeric scores. Finally, open report.md to compare how each model structured its reasoning and whether it provided a formal correctness argument. Exact scores will vary with temperature and prompt phrasing, but the report makes the trade-offs visible.

Next steps

Pipe this harness into a CI job that runs nightly against the latest model versions on Oxlo.ai so you catch regressions immediately. Alternatively, swap the prompt for a long-context agentic task and benchmark DeepSeek V4 Flash against Kimi K2.6 to see how the 1M context window changes the reasoning depth.

DEV Community