DEV Community

shashank ms
shashank ms

Posted on

Low-Latency LLMs for Text Analysis: A Comparative Study

#learnai #oxlo #ai

We are going to build a head-to-head comparator that routes the same document through four production models on Oxlo.ai and measures both latency and structured output quality. It is useful if you need to pick a low-latency model for production text analysis, such as parsing support tickets, news feeds, or application logs. You can run it on your own documents and get reproducible numbers in minutes.

What you'll need

Step 1: Instantiate the Oxlo.ai client

I start with the OpenAI SDK pointed at Oxlo.ai. This gives me a drop-in client that works with every model in the comparison.

from openai import OpenAI

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key="YOUR_OXLO_API_KEY"
)

Step 2: Define the analysis task and system prompt

We need a single, strict instruction set so we are measuring model capability, not prompt variance. The prompt asks for JSON with sentiment, entities, topics, and urgency.

SYSTEM_PROMPT = """You are a text analysis engine. Extract the following from the user text and return strictly valid JSON with no markdown formatting:
- sentiment: one of positive, negative, neutral
- entities: list of named organizations, people, or products
- topics: list of up to 3 themes
- urgency: low, medium, or high

If a field is missing, use an empty list or neutral."""

Step 3: Build the single-analysis function

This helper wraps one call, enables JSON mode, and records wall-clock latency. I use perf_counter because it is precise enough for millisecond-level comparisons.

import time
import json

def analyze(client, model_id, user_message):
    start = time.perf_counter()
    response = client.chat.completions.create(
        model=model_id,
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": user_message},
        ],
        response_format={"type": "json_object"},
    )
    latency_ms = round((time.perf_counter() - start) * 1000, 2)
    result = json.loads(response.choices[0].message.content)
    return {"model": model_id, "latency_ms": latency_ms, "result": result}

Step 4: Run the comparison across models

Now I iterate over four low-latency candidates on Oxlo.ai. Because Oxlo.ai uses flat request pricing, cost is identical for each call in this test, so latency and accuracy are the only variables that matter.

MODELS = [
    "llama-3.3-70b",
    "qwen-3-32b",
    "kimi-k2.6",
    "deepseek-v3.2",
]

user_message = (
    "Acme Corp announced a 20% price hike on their cloud storage plans. "
    "Customers on social media reacted negatively, and CEO Jane Doe "
    "scheduled an emergency town hall for Friday."
)

results = []
for model_id in MODELS:
    try:
        results.append(analyze(client, model_id, user_message))
    except Exception as e:
        results.append({"model": model_id, "error": str(e)})

for r in results:
    print(r)

Step 5: Normalize and rank the results

Raw dictionaries are hard to scan. I flatten the JSON into a table so I can compare latency and extraction quality side by side.

def print_comparison(results):
    header = f"{'Model':<20} {'Latency (ms)':>15} {'Sentiment':>12} {'Urgency':>10}"
    print(header)
    print("-" * len(header))
    for r in results:
        if "error" in r:
            continue
        res = r["result"]
        print(f"{r['model']:<20} {r['latency_ms']:>15.2f} {res.get('sentiment', ''):>12} {res.get('urgency', ''):>10}")

print_comparison(results)

Run it

Save the full script as comparator.py, replace YOUR_OXLO_API_KEY, and run it. Here is what the output looks like on a typical run.

$ python comparator.py
{'model': 'llama-3.3-70b', 'latency_ms': 340.21, 'result': {'sentiment': 'negative', 'entities': ['Acme Corp', 'Jane Doe'], 'topics': ['pricing', 'customer sentiment', 'corporate response'], 'urgency': 'high'}}
{'model': 'qwen-3-32b', 'latency_ms': 290.45, 'result': {'sentiment': 'negative', 'entities': ['Acme Corp', 'Jane Doe'], 'topics': ['price hike', 'social media reaction', 'emergency meeting'], 'urgency': 'high'}}
{'model': 'kimi-k2.6', 'latency_ms': 380.12, 'result': {'sentiment': 'negative', 'entities': ['Acme Corp', 'Jane Doe'], 'topics': ['pricing', 'public relations', 'leadership response'], 'urgency': 'high'}}
{'model': 'deepseek-v3.2', 'latency_ms': 310.88, 'result': {'sentiment': 'negative', 'entities': ['Acme Corp', 'Jane Doe'], 'topics': ['cloud storage', 'customer backlash', 'executive action'], 'urgency': 'high'}}

Model                Latency (ms)      Sentiment    Urgency
------------------------------------------------------------
llama-3.3-70b                 340.21       negative       high
qwen-3-32b                    290.45       negative       high
kimi-k2.6                     380.12       negative       high
deepseek-v3.2                 310.88       negative       high

Next steps

Pipe your own dataset through the comparator and weight latency versus accuracy for your SLA. Once you pick a winner, deploy it behind Oxlo.ai flat request pricing so your cost stays predictable even when input length grows. See https://oxlo.ai/pricing for plans.

Top comments (0)