shashank ms

Posted on Aug 17

Proprietary vs Open-Source LLM Models: A Comprehensive Guide

#learnai #oxlo #ai

We are building a support ticket router that classifies incoming questions and sends them to either an open-source or a proprietary model depending on complexity. This saves money on easy tickets while preserving quality for hard ones. Teams running high-volume support queues will see the biggest impact.

What you'll need

An Oxlo.ai API key from https://portal.oxlo.ai
Python 3.10 or newer
The OpenAI SDK: pip install openai

Step 1: Configure the Oxlo.ai client

Oxlo.ai is fully OpenAI SDK compatible, so we initialize one client and reuse it for every model. Point the base URL at Oxlo.ai and set your API key.

from openai import OpenAI
import os

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key=os.environ.get("OXLO_API_KEY", "YOUR_OXLO_API_KEY")
)

Step 2: Build the ticket classifier

We need a fast judge that labels each ticket as standard or complex. I use DeepSeek V3.2 because it is efficient, handles reasoning well, and sits on Oxlo.ai's free tier. The classifier returns strict JSON so we can parse it without regex.

import json

ROUTER_PROMPT = """You are a routing engine.
Read the support ticket and classify it.
Return ONLY a JSON object with this exact shape:
{"tier": "standard"} or {"tier": "complex"}.
Choose "complex" if the ticket involves debugging, architecture, security, or ambiguous failures. Choose "standard" for pricing, billing, password resets, or feature requests."""

def classify_ticket(ticket_text: str) -> str:
    response = client.chat.completions.create(
        model="deepseek-v3.2",
        messages=[
            {"role": "system", "content": ROUTER_PROMPT},
            {"role": "user", "content": ticket_text},
        ],
        response_format={"type": "json_object"},
    )
    result = json.loads(response.choices[0].message.content)
    return result.get("tier", "standard")

Step 3: Define agent prompts for each tier

This is where the proprietary versus open-source trade-off becomes concrete. For standard tickets, Llama 3.3 70B is fast, open, and more than capable. For complex tickets, Kimi K2.6 brings deeper reasoning and longer context. Both are available on Oxlo.ai with the same flat per-request pricing, so we are choosing based on capability, not token math.

SYSTEM_PROMPT_STANDARD = """You are a concise support agent.
Answer the user's question in one short paragraph.
Be friendly, but do not ask follow-up questions."""

SYSTEM_PROMPT_COMPLEX = """You are a senior solutions architect.
Investigate the issue thoroughly. List likely root causes, suggest concrete remediation steps, and ask for only the most critical missing log or metric.
Use bullet points for clarity."""

Step 4: Build the tiered responder

The responder maps the tier label to the right model ID and system prompt, then calls Oxlo.ai. Because every request costs the same flat amount on Oxlo.ai, we can mix and match tiers without worrying about input length blowing up the bill.

TIER_CONFIG = {
    "standard": {
        "model": "llama-3.3-70b",
        "prompt": SYSTEM_PROMPT_STANDARD,
    },
    "complex": {
        "model": "kimi-k2.6",
        "prompt": SYSTEM_PROMPT_COMPLEX,
    },
}

def draft_reply(ticket_text: str, tier: str) -> str:
    config = TIER_CONFIG[tier]
    response = client.chat.completions.create(
        model=config["model"],
        messages=[
            {"role": "system", "content": config["prompt"]},
            {"role": "user", "content": ticket_text},
        ],
    )
    return response.choices[0].message.content

Step 5: Wire up the orchestrator

Finally, we chain the pieces together. The orchestrator prints which tier and model handled the ticket so we can audit the routing later.

def handle_ticket(ticket_text: str) -> dict:
    tier = classify_ticket(ticket_text)
    reply = draft_reply(ticket_text, tier)
    model_used = TIER_CONFIG[tier]["model"]
    return {
        "tier": tier,
        "model": model_used,
        "reply": reply,
    }

Run it

Here is a small batch of tickets. When I run this script, the classifier sends the password question to Llama 3.3 70B and the Kubernetes incident to Kimi K2.6.

if __name__ == "__main__":
    tickets = [
        "How do I reset my password? I clicked the link but never got the email.",
        "Our Kubernetes cluster is failing with intermittent 503s after the latest cert-manager update. The ingress-nginx logs show TLS handshake errors. We are on EKS 1.29.",
        "Do you offer academic discounts for teams under 10 seats?",
    ]

    for t in tickets:
        result = handle_ticket(t)
        print(f"\n--- Ticket ---\n{t}")
        print(f"--- Routed to {result['tier']} ({result['model']}) ---")
        print(result["reply"])

Example output:

--- Ticket ---
How do I reset my password? I clicked the link but never got the email.
--- Routed to standard (llama-3.3-70b) ---
Check your spam or promotions folder first. If it is not there, try whitelisting noreply@company.com and request another reset from the login page.

--- Ticket ---
Our Kubernetes cluster is failing with intermittent 503s after the latest cert-manager update. The ingress-nginx logs show TLS handshake errors. We are on EKS 1.29.
--- Routed to complex (kimi-k2.6) ---
Likely root causes:
- cert-manager may have issued a certificate with an incompatible key algorithm or SAN list that ingress-nginx rejects.
- The cert-manager CRDs might not match the controller version after the update.

Remediation steps:
1. Verify the Certificate object status: kubectl describe certificate tls-secret -n ingress-nginx.
2. Check ingress-nginx controller logs for the exact TLS error string.
3. Roll back cert-manager to the previous version if the CRD migration was skipped.

Please share the output of kubectl get events -n cert-manager from the last 10 minutes.

--- Ticket ---
Do you offer academic discounts for teams under 10 seats?
--- Routed to standard (llama-3.3-70b) ---
Yes, we offer a 50% academic discount for verified educational teams under 10 seats. Apply from your billing dashboard under Plan > Academic Verification.

Wrap-up and next steps

This router demonstrates the practical difference between open-source and proprietary models. Oxlo.ai makes the comparison easy because both categories live on the same flat per-request pricing layer, so you are not penalized for sending a long log dump to a proprietary model or for bouncing ten quick questions through an open-source one. You can see current plan details at https://oxlo.ai/pricing.

Two concrete next steps. First, add a feedback loop: store the ticket, tier, and a thumbs-up or thumbs-down, then use that data to fine-tune the router prompt. Second, expose the agent as an async FastAPI endpoint so it can sit behind your existing support intake form and stream responses back with Oxlo.ai's streaming support.

DEV Community