DEV Community

shashank ms
shashank ms

Posted on

Introduction to LLM Optimization

#learnai #oxlo #ai

We are going to build a request optimizer that compresses conversation history, classifies incoming queries, and routes them to the most efficient Oxlo.ai model for the job. It cuts latency and keeps output quality high by matching task complexity to model capability. If you run workloads with long contexts or mixed request types, this pattern saves both time and money on any provider, and it pairs especially well with Oxlo.ai's flat per-request pricing (https://oxlo.ai/pricing).

What you'll need

Step 1: Initialize the Oxlo.ai client

I start every project with a thin wrapper around the OpenAI SDK so I can swap providers later without touching business logic. Point the base URL at Oxlo.ai and load your key.

from openai import OpenAI

client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")

Step 2: Compress conversation history

Long chat logs slow down inference and dilute attention. I use a fast model to distill old messages into a single context paragraph. On Oxlo.ai this costs the same flat rate regardless of input length, but compression still improves latency and focus.

COMPRESSION_PROMPT = """You are a context compressor. Summarize the provided conversation history into one concise paragraph. Preserve all facts, user preferences, and open tasks. Omit filler words and greetings."""
def compress_history(history: str) -> str:
    if not history.strip():
        return ""
    response = client.chat.completions.create(
        model="deepseek-v3.2",
        messages=[
            {"role": "system", "content": COMPRESSION_PROMPT},
            {"role": "user", "content": history},
        ],
    )
    return response.choices[0].message.content

Step 3: Classify query complexity

Not every question needs a 70B parameter model. I run a lightweight classifier with JSON mode so I can route simple FAQs to a general model and coding questions to a specialized one.

CLASSIFIER_PROMPT = """You are a query classifier. Analyze the user message and return a JSON object with exactly two keys: "intent" and "confidence". intent must be one of "simple", "coding", or "complex". confidence is a float between 0 and 1."""
import json

def classify_query(user_message: str) -> dict:
    response = client.chat.completions.create(
        model="deepseek-v3.2",
        messages=[
            {"role": "system", "content": CLASSIFIER_PROMPT},
            {"role": "user", "content": user_message},
        ],
        response_format={"type": "json_object"},
    )
    return json.loads(response.choices[0].message.content)

Step 4: Route to the optimal model

I map intents to Oxlo.ai models based on their strengths. Llama 3.3 70B handles general chat, Kimi K2.6 takes coding tasks, and Qwen 3 32B manages complex reasoning.

MODEL_MAP = {
    "simple": "llama-3.3-70b",
    "coding": "kimi-k2.6",
    "complex": "qwen-3-32b",
}

def pick_model(intent: str) -> str:
    return MODEL_MAP.get(intent, "llama-3.3-70b")

Step 5: Assemble the optimizer agent

Now I wire the pieces together. The agent compresses any existing history, classifies the new message, selects the model, and returns the result along with metadata so I can audit routing decisions later.

The agent system prompt:

AGENT_PROMPT = """You are a helpful technical support agent. Answer concisely. If the user asks about code, include working snippets. If context is provided, reference it explicitly."""
def optimize_and_respond(user_message: str, history: str = "") -> dict:
    context = compress_history(history) if history else ""
    classification = classify_query(user_message)
    model = pick_model(classification["intent"])

    full_input = user_message
    if context:
        full_input = f"Context from previous conversation: {context}\n\nNew query: {user_message}"

    response = client.chat.completions.create(
        model=model,
        messages=[
            {"role": "system", "content": AGENT_PROMPT},
            {"role": "user", "content": full_input},
        ],
    )

    return {
        "model_used": model,
        "intent": classification["intent"],
        "confidence": classification["confidence"],
        "answer": response.choices[0].message.content,
    }

Run it

I test with a long, meandering history and a crisp coding request. The optimizer should compress the noise, detect the coding intent, and route to Kimi K2.6.

if __name__ == "__main__":
    long_history = (
        "User: Hi there, I need help with Python.\n"
        "Agent: Hello! Sure, what do you need?\n"
        "User: I am trying to parse JSON but I keep getting errors.\n"
        "Agent: Can you share the snippet?\n"
        "User: Here it is: json.loads(data). It says Expecting property name.\n"
        "Agent: That usually means a trailing comma. Check your JSON file.\n"
        "User: Ah fixed, thanks. Now I also want to know how to optimize LLM calls.\n"
    )

    result = optimize_and_respond(
        user_message="Write me a Python function that rewrites prompts to be shorter.",
        history=long_history
    )

    print(f"Model routed: {result['model_used']}")
    print(f"Intent: {result['intent']} ({result['confidence']})")
    print("---")
    print(result["answer"])

Example output:

Model routed: kimi-k2.6
Intent: coding (0.94)
---
Here is a concise function that strips filler words and enforces a word limit.



```python
def compress_prompt(text: str, max_words: int = 50) -> str:
    words = text.split()
    if len(words) <= max_words:
        return text
    return " ".join(words[:max_words]) + "..."
```

Next steps

Cache compressed summaries in Redis so repeated queries from the same user skip the summarization step entirely. You could also add a retry wrapper with exponential backoff around the classifier to handle the rare malformed JSON parse. Both improvements are straightforward additions to the pattern above.

Top comments (0)