We're going to build a completion ranking pipeline that takes a text prompt, generates continuations from three different Oxlo.ai models, and uses a judge model to pick the best one. This saves time when you need to decide which model fits your domain without running manual benchmarks.
What you'll need
- Python 3.10 or newer
- The OpenAI SDK:
pip install openai - An Oxlo.ai API key from https://portal.oxlo.ai
Step 1: Configure the Oxlo.ai client
First, I set up the client. Oxlo.ai is fully OpenAI SDK compatible, so I only need to swap the base URL and plug in my key. I test it with a single call to Llama 3.3 70B to confirm the connection works.
from openai import OpenAI
import os
client = OpenAI(
base_url="https://api.oxlo.ai/v1",
api_key=os.environ.get("OXLO_API_KEY")
)
response = client.chat.completions.create(
model="llama-3.3-70b",
messages=[
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Say 'Connection OK' and nothing else."},
],
)
print(response.choices[0].message.content)Step 2: Generate completions from multiple models
Next, I write a helper that fires the same prompt to three models in parallel. Because Oxlo.ai uses flat per-request pricing, running Llama 3.3 70B, Qwen 3 32B, and DeepSeek V3.2 together stays predictable even when completions get long.
from concurrent.futures import ThreadPoolExecutor, as_completed
MODELS = [
"llama-3.3-70b",
"qwen-3-32b",
"deepseek-v3.2",
]
def generate_completion(model, prompt, system="You are a helpful assistant."):
resp = client.chat.completions.create(
model=model,
messages=[
{"role": "system", "content": system},
{"role": "user", "content": prompt},
],
temperature=0.7,
max_tokens=256,
)
return {
"model": model,
"text": resp.choices[0].message.content.strip()
}
def generate_all(prompt):
results = {}
with ThreadPoolExecutor(max_workers=3) as ex:
futures = {ex.submit(generate_completion, m, prompt): m for m in MODELS}
for future in as_completed(futures):
data = future.result()
results[data["model"]] = data["text"]
return resultsStep 3: Build the judge prompt
I need a judge to evaluate the outputs. I use Kimi K2.6 because it handles advanced reasoning and long context well. The judge receives the original prompt and all completions, then ranks them by coherence, relevance, and fluency.
Here is the system prompt I use for the judge agent.
JUDGE_SYSTEM_PROMPT = """You are an expert evaluator of language model outputs.
Your job is to rank completions from best to worst based on:
1. Coherence with the prompt
2. Grammatical fluency
3. Factual consistency
Respond with a JSON object containing:
- rankings: a list of model names from best to worst
- reasoning: a one-sentence explanation for the top choice"""Step 4: Rank outputs with Kimi K2.6
Now I send the completions to the judge. I format the user message so the judge sees the original prompt and each model's output clearly labeled. I set the temperature low to keep the evaluation deterministic.
import json
def rank_completions(prompt, completions):
formatted = f"Original prompt: {prompt}\n\n"
for model, text in completions.items():
formatted += f"--- {model} ---\n{text}\n\n"
resp = client.chat.completions.create(
model="kimi-k2.6",
messages=[
{"role": "system", "content": JUDGE_SYSTEM_PROMPT},
{"role": "user", "content": formatted},
],
temperature=0.2,
max_tokens=512,
)
raw = resp.choices[0].message.content.strip()
if raw.startswith("
```"):
raw = raw.split("```
")[1].replace("json", "").strip()
return json.loads(raw)Step 5: Wire the full pipeline
Finally, I tie everything together in a small CLI script. It reads a prompt, gathers completions, calls the judge, and prints the winner along with all outputs for inspection.
def main():
prompt = input("Enter a prompt: ").strip()
if not prompt:
print("Empty prompt. Exiting.")
return
print("\nGenerating completions...")
completions = generate_all(prompt)
print("Judging...")
result = rank_completions(prompt, completions)
print("\n=== Rankings ===")
for i, model in enumerate(result["rankings"], 1):
print(f"{i}. {model}")
print(f"\nTop pick reasoning: {result['reasoning']}")
print("\n=== Full outputs ===")
for model, text in completions.items():
print(f"\n{model}:\n{text}")
if __name__ == "__main__":
main()Run it
Save the script as rank_completions.py, export your key, and run it. Below is an example session using a technical prompt.
export OXLO_API_KEY="your-key-here"
python rank_completions.pyExample output:
Enter a prompt: The transformer architecture revolutionized NLP by
Generating completions...
Judging...
=== Rankings ===
1. llama-3.3-70b
2. qwen-3-32b
3. deepseek-v3.2
Top pick reasoning: Llama 3.3 70B provided the most coherent and technically accurate continuation of the transformer concept.
=== Full outputs ===
llama-3.3-70b:
introducing self-attention mechanisms that process entire sequences in parallel, eliminating the recurrence bottlenecks of RNNs and enabling scalable pre-training on massive corpora.
qwen-3-32b:
enabling models to capture long-range dependencies through attention mechanisms, which allows parallel computation and significantly improves training efficiency over previous sequential models.
deepseek-v3.2:
using attention mechanisms to weigh input tokens dynamically, which improved parallelization and reduced training time compared to recurrent networks.Next steps
Add a local SQLite cache that stores the winning model for each prompt hash. After a few runs, you can skip the judge and route common prompts straight to the model that historically performs best for your domain.
Alternatively, swap the judge model for GLM 5 or DeepSeek R1 671B if you need deeper reasoning for complex coding or agentic completions.
Top comments (0)