DEV Community: shashank ms

Using LLM for Text Generation Tasks

shashank ms — Tue, 07 Jul 2026 03:33:58 +0000

We are going to build a batch product description generator that turns a CSV of sparse product specs into polished, ready-to-publish marketing copy. This is for e-commerce ops teams who are tired of writing the same blurbs over and over. I will use Oxlo.ai because its request-based pricing means I can feed it long, messy supplier spec sheets without watching the meter spin on every extra token.

What you'll need

Python 3.10 or newer
An Oxlo.ai API key from https://portal.oxlo.ai
The OpenAI SDK: pip install openai
Pandas for CSV handling: pip install pandas

Step 1: Configure the Oxlo.ai client

I start by importing the OpenAI SDK and pointing it at Oxlo.ai. Because Oxlo.ai is fully OpenAI API compatible, this is a drop-in replacement. No custom adapters needed.

from openai import OpenAI
import os

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key=os.environ.get("OXLO_API_KEY")
)

I keep the key in an environment variable so I do not accidentally commit it.

Step 2: Craft the system prompt

The system prompt is the only part of the agent I will tune heavily. It defines voice, length, and what to avoid. I treat it as a config file and keep it in a constant so I can iterate without touching the logic.

SYSTEM_PROMPT = """You are a senior e-commerce copywriter. Your job is to turn raw product specifications into compelling product descriptions.

Rules:
- Write exactly one paragraph of 40 to 60 words.
- Highlight the single most important benefit first.
- Use active voice and concrete nouns.
- Do not use exclamation points or all-caps.
- If technical specs are provided, translate one key spec into a customer benefit.

Output only the description text. No markdown, no preamble."""

Step 3: Build the generation function

Next I wrap the API call in a small function that accepts a raw spec string and returns the generated copy. I use Llama 3.3 70B because it is Oxlo.ai's general-purpose flagship and handles marketing voice reliably. I also add a short sleep to be polite if I scale this up later.

import time

def generate_description(product_specs: str, model: str = "llama-3.3-70b") -> str:
    response = client.chat.completions.create(
        model=model,
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": product_specs},
        ],
        temperature=0.7,
        max_tokens=256,
    )
    time.sleep(0.5)
    return response.choices[0].message.content.strip()

Step 4: Load data and run the batch

For this project I assume a CSV with two columns: sku and raw_specs. I read it with Pandas, apply the generator row by row, and append the result to a new column. This pattern works for hundreds of SKUs. If I needed thousands I would add a simple retry loop around the API call.

import pandas as pd

df = pd.read_csv("products.csv")

descriptions = []
for _, row in df.iterrows():
    desc = generate_description(row["raw_specs"])
    descriptions.append(desc)

df["generated_description"] = descriptions
df.to_csv("products_with_copy.csv", index=False)

print(f"Generated {len(descriptions)} descriptions.")

I keep the loop explicit instead of using apply so I can add logging or error handling later without refactoring.

Step 5: Inspect the output

After the run I always spot-check a few rows. Here is a quick script to pretty-print a sample so I can scan for tone drift or hallucinated claims before the copy goes live.

for i in range(min(3, len(df))):
    print(f"SKU: {df.loc[i, 'sku']}")
    print(f"Specs: {df.loc[i, 'raw_specs']}")
    print(f"Copy:  {df.loc[i, 'generated_description']}")
    print("-" * 40)

Run it

To test the whole flow without a CSV, I can call the generator directly with a raw string. This is the same call the batch loop makes under the hood.

test_specs = """
SKU: BK-2024-009
Name: Terra Hiking Backpack
Material: 420D recycled nylon
Capacity: 28L
Features: Hydration sleeve, rain cover, ventilated back panel
"""

print(generate_description(test_specs))

Example output:

The Terra Hiking Backpack keeps you moving with a ventilated back panel that cuts heat on steep ascents. Built from 420D recycled nylon, it offers a 28-liter capacity with a dedicated hydration sleeve and integrated rain cover so you are ready when weather turns. Lightweight, durable, and designed for the trail.

Next steps

This agent works, but it is just a starting point. Two concrete upgrades I would ship next:

Add structured output by switching to JSON mode and returning description, bullet_points, and seo_title in one request. Oxlo.ai supports this on all chat models.
Pipe the output directly into your storefront or PIM API so the copy moves from generated to published without a manual CSV handoff.

If your product specs are long or your agent needs multi-turn reasoning, Oxlo.ai's request-based pricing stays flat no matter how much context you stuff into the prompt. You can explore plans and model options at https://oxlo.ai/pricing.

LLM Model Quantization Explained

shashank ms — Tue, 07 Jul 2026 01:32:07 +0000

Large language models are memory-bound. A 70 billion parameter model in FP32 requires roughly 280 GB of GPU memory just for weights, before accounting for activations, gradients, or the KV cache. Quantization compresses these weights into lower-precision representations, making it feasible to serve advanced models on commodity hardware. For developers, understanding quantization is no longer optional. It directly shapes latency, throughput, and the hardware bill behind every API call.

What Is Quantization?

At its core, quantization is a compression technique that maps high-precision floating-point numbers to lower-precision integers or floats. The most common jump

Integrating LLM with Speech Recognition

shashank ms — Mon, 06 Jul 2026 23:37:53 +0000

Building voice-enabled applications usually means stitching together a transcription service, an LLM provider, and sometimes a text-to-speech engine. Each hop adds latency, cost accounting complexity, and another set of credentials to rotate. Oxlo.ai simplifies this stack by exposing both audio transcription and chat inference behind a single OpenAI-compatible endpoint, with request-based pricing that removes the token-cost penalty on long audio transcripts.

The Standard Pipeline

Most voice-to-insight systems follow a three-stage pattern. First, an automatic speech recognition (ASR) model converts raw audio into text. Second, an LLM processes that text to summarize, extract entities, or reason about next actions. Third, an optional text-to-speech (TTS) model turns the response back into audio for conversational interfaces.

Oxlo.ai hosts models for every stage. Whisper Large v3 handles transcription with strong accuracy across accents and noisy environments. Llama 3.3 70B, Qwen 3 32B, DeepSeek R1 671B MoE, and Kimi K2.6 provide reasoning, coding, and multilingual understanding over the resulting text. Kokoro 82M offers lightweight TTS when you need voice output. Because every model sits behind the same base URL and SDK, you can move from audio file to spoken response without leaving the Oxlo.ai stack.

End-to-End Code

The OpenAI SDK drops in directly. Set base_url to https://api.oxlo.ai/v1 and point your existing transcription and chat code at Oxlo.ai models.

import os
from openai import OpenAI

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key=os.environ["OXLO_API_KEY"]
)

# 1. Transcribe audio with Whisper Large v3
with open("recording.wav", "rb") as audio_file:
    transcription = client.audio.transcriptions.create(
        model="Whisper Large v3",
        file=audio_file
    )

# 2. Summarize and extract action items with Llama 3.3 70B
chat = client.chat.completions.create(
    model="Llama 3.3 70B",
    messages=[
        {
            "role": "system",
            "content": "You are a meeting assistant. Summarize the transcript and list action items."
        },
        {
            "role": "user",
            "content": transcription.text
        }
    ],
    response_format={"type": "json_object"}
)

print(chat.choices[0].message.content)

This single-client pattern keeps credential management simple and eliminates cross-provider networking overhead. If you need voice output, you can call Kokoro 82M through the same client.audio.speech.create interface.

Long Context and Cost

Transcripts from hour-long meetings, podcasts, or call center recordings can easily exceed tens of thousands of tokens. On token-based providers, that input length drives cost up linearly. Oxlo.ai uses request-based pricing: one flat cost per API call regardless of how long the transcript is. For transcription-to-LLM workflows, this means you can pass the full raw text into a model like DeepSeek V4 Flash with its 1 million token context window, or Kimi K2.6 with 131K context, without watching the meter run on every additional paragraph.

If your workload involves agentic loops, where the LLM iteratively reasons over the transcript and calls tools, the savings compound. You pay per request, not per token consumed across multiple reasoning steps.

Structured Output and Tool Use

Raw transcripts are rarely the final product. You usually need structured data: speaker labels, sentiment scores, or extracted calendar events. Oxlo.ai supports JSON mode and function calling on its chat models, so you can constrain the LLM to emit valid JSON or invoke external tools directly from the transcript content.

For example, after transc

Using LLM for Text Analysis

shashank ms — Mon, 06 Jul 2026 23:35:16 +0000

We are building a feedback analysis agent that turns raw, unstructured customer comments into structured JSON. It helps product and support teams spot urgent issues and trending topics without reading every ticket manually.

What you'll need

Python 3.10 or newer
The OpenAI SDK: pip install openai
An Oxlo.ai API key from https://portal.oxlo.ai

Step 1: Configure the Oxlo.ai client

I start every project by verifying the client can reach the API. This snippet initializes the OpenAI SDK pointing at Oxlo.ai and sends a smoke-test request.

from openai import OpenAI

client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")

# Quick connectivity check
response = client.chat.completions.create(
    model="llama-3.3-70b",
    messages=[{"role": "user", "content": "ping"}],
)
print(response.choices[0].message.content)

Step 2: Write the analysis system prompt

The system prompt is the only training the agent gets. I keep it strict about output format so downstream code never has to guess.

SYSTEM_PROMPT = """You are a text-analysis assistant. Your job is to read a piece of customer feedback and return a single JSON object with exactly these keys:

- sentiment: one of "positive", "neutral", or "negative"
- topics: an array of up to three topics, e.g. ["billing", "onboarding"]
- urgency: one of "low", "medium", or "high"
- summary: a one-sentence summary of the feedback

Rules:
1. Output ONLY valid JSON. No markdown, no explanation.
2. If the text is not feedback, set sentiment to "neutral", topics to [], urgency to "low", and summarize anyway."""

Step 3: Create the analysis function with JSON mode

With the prompt locked, I wrap the call in a small function. I use JSON mode so the model is constrained to valid output, which saves me from writing a retry parser.

import json

def analyze_feedback(text: str) -> dict:
    response = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": text},
        ],
        response_format={"type": "json_object"},
        temperature=0.1,
    )
    raw = response.choices[0].message.content
    return json.loads(raw)

Step 4: Process a batch of feedback records

Real data never arrives one line at a time. I loop over a list of raw strings, collect the structured results, and write them to a JSON Lines file for later analysis.

FEEDBACK_BATCH = [
    "I love the new dashboard, but exporting CSVs is still broken. Please fix it soon.",
    "Your onboarding wizard made setup trivial. Great work.",
    "I was charged twice this month and your refund page gives a 404. This is unacceptable.",
    "Meh. It works most days.",
]

with open("feedback_analysis.jsonl", "w") as f:
    for item in FEEDBACK_BATCH:
        result = analyze_feedback(item)
        record = {"input": item, "analysis": result}
        f.write(json.dumps(record) + "\n")
        print(record)

Step 5: Aggregate and display a summary report

Once the structured data is on disk, a second pass counts topics and flags anything marked high urgency. This is where the ROI shows up.

from collections import Counter

urgent_items = []
topic_counts = Counter()

with open("feedback_analysis.jsonl", "r") as f:
    for line in f:
        record = json.loads(line)
        analysis = record["analysis"]
        topic_counts.update(analysis.get("topics", []))
        if analysis.get("urgency") == "high":
            urgent_items.append(record["input"])

print("Topic breakdown:", dict(topic_counts))
print(f"High-urgency items: {len(urgent_items)}")
for item in urgent_items:
    print("-", item)

Run it

Save everything in analyze.py, set your API key, and run python analyze.py. The first pass writes feedback_analysis.jsonl; the second pass prints a summary like this:

Topic breakdown: {'billing': 1, 'export': 1, 'onboarding': 1, 'reliability': 1}
High-urgency items: 1
- I was charged twice this month and your refund page gives a 404. This is unacceptable.

Next steps

To productionize this, pipe the JSON Lines into a TinyDB or SQLite table so you can query trends over time. If your feedback is multilingual, swap the model to qwen-3-32b on Oxlo.ai and keep the same prompt. Because Oxlo.ai uses per-request pricing, long feedback threads do not inflate your bill the way token-based metering would. See https://oxlo.ai/pricing for details.

Deploying LLM Models on Mobile Devices

shashank ms — Mon, 06 Jul 2026 23:33:17 +0000

Running large language models on mobile devices is one of the hardest optimization problems in modern AI engineering. Phones have limited RAM, strict thermal budgets, and battery constraints that make hosting a multi-billion parameter model locally impractical for most applications. Yet users expect low-latency, intelligent experiences whether they are online or offline. The solution is usually a tiered architecture that combines quantized on-device models with a reliable cloud inference backend.

The Mobile LLM Landscape

On-device inference has matured with frameworks like Core ML, TensorFlow Lite, and MLX Swift. You can now run 1B to 4B parameter models on flagship devices with acceptable latency for autocomplete or classification tasks. Anything larger, such as Llama 3.3 70B or DeepSeek R1 671B MoE, remains firmly in the data center. Most production mobile apps therefore use a hybrid pattern: a small local model for offline resilience and sensitive data processing, and a cloud API for heavy reasoning, coding, or vision tasks.

Architectural Patterns for Production Apps

Three patterns dominate. First, fully on-device for privacy-critical features like local transcription or on-phone embedding generation. Second, fully cloud-backed for agentic workflows, code generation, or multimodal chat. Third, and most common, a routing layer that sends simple queries to a 2B parameter local model and escalates complex prompts to a cloud provider. This minimizes latency for common tasks while preserving capability for edge cases.

Integrating Cloud Inference with Oxlo.ai

This is where Oxlo.ai becomes a strong fit. Because Oxlo.ai charges a flat cost per API request rather than per token, mobile apps with long system prompts, large context windows, or multi-turn conversations avoid the unpredictable billing that scales with input length. For a mobile client that might send thousands of tokens of conversation history with every tap, request-based pricing keeps costs predictable.

Oxlo.ai is fully OpenAI SDK compatible, so integrating it into an iOS or Android project requires no custom networking layer. You point the base URL to https://api.oxlo.ai/v1 and use your existing OpenAI client. Below is a minimal Swift example using URLSession.

import Foundation

final class OxloAIClient {
    private let apiKey: String
    private let session = URLSession.shared
    private let baseURL = URL(string: "https://api.oxlo.ai/v1/chat/completions")!

    init(apiKey: String) {
        self.apiKey = apiKey
    }

    func complete(
        messages: [[String: String]],
        model: String = "llama-3.3-70b",
        completion: @escaping (Result<Data, Error>) -> Void
    ) {
        var request = URLRequest(url: baseURL)
        request.httpMethod = "POST"
        request.setValue("application/json", forHTTPHeaderField: "Content-Type")
        request.setValue("Bearer \(apiKey)", forHTTPHeaderField: "Authorization")

        let body: [String: Any] = [
            "model": model,
            "messages": messages,
            "stream": false
        ]

        do {
            request.httpBody = try JSONSerialization.data(withJSONObject: body)
        } catch {
            completion(.failure(error))
            return
        }

        let task = session.dataTask(with: request) { data, _, error in
            if let error = error {
                completion(.failure(error))
            } else if let data = data {
                completion(.success(data))
            }
        }
        task.resume()
    }
}

Because Oxlo.ai exposes a fully OpenAI-compatible API at https://api.oxlo.ai/v1, you can use the official OpenAI Swift SDK or any HTTP client. The only change is the base URL and API key.

Why Request Pricing Matters for Mobile

Mobile UX patterns naturally generate long prompts. Every message in a chat interface appends to the conversation history. Agentic features might include large tool definitions or retrieved document chunks. Under token-based pricing, each user tap gets more expensive as the session deepens. Oxlo.ai flattens this curve. A single request costs the same whether it carries a 512-token greeting or a 16,000-token context dump. For product teams building conversational or RAG-heavy mobile apps, this predictability simplifies unit economics. See https://oxlo.ai/pricing for plan details.

Selecting Models for Mobile Backends

Not every mobile feature needs a 400B parameter model. Oxlo.ai offers a spectrum. For fast categorization or routing, you might call Oxlo.ai Coder Fast or a lightweight Qwen 3 variant. For deep reasoning or code review, route to DeepSeek R1 671B MoE or Kimi K2.6. For vision features, use Gemma 3 27B or Kimi VL A3B. Because Oxlo.ai has no cold starts on popular models, the user never waits for a GPU to spin up, which is critical on mobile where patience is measured in milliseconds.

The On-Device Piece of the Puzzle

For the local side of the architecture, quantization is non-negotiable. Use GGUF Q4_K_M or Core ML int8 palettes to compress a 3B or 4B model into a 2GB footprint that fits within a mobile app bundle. Frameworks like llama.cpp and MLX Swift let you offload memory management and KV-cache optimization to mature engines. Keep the local model responsible for intent classification, PII stripping, or offline caching. When the local router decides the query is too complex, it forwards to Oxlo.ai.

Putting It All Together

A production-grade mobile LLM stack typically looks like this: the user types a message; a 2B parameter on-device classifier scores complexity and privacy level; simple queries get an instant local reply; everything else hits Oxlo.ai over HTTPS with streaming enabled. This gives you offline resilience, data sovereignty for sensitive inputs, and unlimited capability via the cloud. Oxlo.ai's flat per-request pricing and OpenAI-compatible endpoints remove the usual backend friction, letting mobile engineers focus on the client rather than infrastructure.

Deploying LLMs on mobile is not about choosing between edge and cloud. It is about intelligently routing between them. Oxlo.ai provides the server-side half of that equation with predictable request-based pricing, broad model coverage, and zero cold-start latency. If you are building the next generation of mobile AI features, start with a quantized local guardian and a robust Oxlo.ai backend.

LLM Model Pruning: Techniques and Best Practices

shashank ms — Mon, 06 Jul 2026 21:37:19 +0000

Model pruning cuts inference costs by removing redundant weights, but hand-picking which layers to drop is tedious and error-prone. I built a lightweight pruning workbench that uses Oxlo.ai to recommend a pruning strategy and judge output quality, while the actual weight surgery happens locally on a small open model. Because Oxlo.ai uses flat per-request pricing, running a dozen evaluator calls with full context does not inflate the bill the way token-based providers do.

What you'll need

Python 3.10+
pip install openai torch transformers datasets
An Oxlo.ai API key from https://portal.oxlo.ai
About 1 GB of free disk space for the 135 M parameter test model

Step 1: Set up the Oxlo.ai client and local environment

I start by importing the libraries and initializing the OpenAI-compatible client pointing at Oxlo.ai. I also define the local test model and the sparsity target we will apply.

import json
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from openai import OpenAI

client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")

MODEL_ID = "HuggingFaceTB/SmolLM-135M"
TARGET_SPARSITY = 0.30

Step 2: Inspect the model and generate a pruning plan

I load the 135 M parameter model and extract its layer configuration. Instead of guessing which layers to prune, I send the architecture summary to Llama 3.3 70B on Oxlo.ai and ask for a structured plan. Here is the system prompt I use for the pruning strategist.

SYSTEM_PROMPT = """You are a senior ML engineer specializing in model compression. The user will provide a transformer architecture summary and a target sparsity ratio. Respond with a JSON object containing a single key \"prune_layers\" whose value is a list of integer layer indices to prune. Prefer pruning later MLP layers over early attention layers. Respond with only the JSON object."""

Next, I build the summary and call Oxlo.ai.

tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
model = AutoModelForCausalLM.from_pretrained(MODEL_ID, torch_dtype=torch.float32)

num_layers = len(model.model.layers)
hidden_size = model.config.hidden_size
intermediate_size = model.config.intermediate_size

arch_summary = (
    f"Model: {MODEL_ID}\n"
    f"Layers: {num_layers}\n"
    f"Hidden size: {hidden_size}\n"
    f"Intermediate size: {intermediate_size}\n"
    f"Target sparsity: {int(TARGET_SPARSITY * 100)}%"
)

response = client.chat.completions.create(
    model="llama-3.3-70b",
    messages=[
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user", "content": arch_summary},
    ],
)

plan = json.loads(response.choices[0].message.content)
layers_to_prune = plan["prune_layers"]
print("Layers selected for pruning:", layers_to_prune)

Step 3: Apply structured pruning to the local model

I clone the original model so I can compare later. For each layer index the agent selected, I compute the L2 norm of every neuron in gate_proj and zero out the weakest 30 percent. I propagate the same mask to up_proj and down_proj to keep dimensions consistent.

import copy

pruned_model = copy.deepcopy(model)

def prune_mlp_layers(target_model, layer_indices, sparsity_ratio):
    layers = target_model.model.layers
    for idx in layer_indices:
        if idx < 0 or idx >= len(layers):
            continue
        mlp = layers[idx].mlp
        gate = mlp.gate_proj.weight.data
        norms = torch.norm(gate, dim=1)
        k = int(sparsity_ratio * len(norms))
        if k == 0:
            continue
        threshold = torch.kthvalue(norms, k).values
        mask = norms > threshold
        gate[~mask] = 0
        mlp.up_proj.weight.data[~mask] = 0
        mlp.down_proj.weight.data[:, ~mask] = 0
    return target_model

pruned_model = prune_mlp_layers(pruned_model, layers_to_prune, TARGET_SPARSITY)
print(f"Pruned {len(layers_to_prune)} layers at {TARGET_SPARSITY:.0%} sparsity.")

Step 4: Evaluate outputs with Oxlo.ai as a judge

I generate answers from both the original and pruned models on a small validation set. Then I send each pair to Oxlo.ai for a side-by-side quality score. Because Oxlo.ai pricing is per request, I can include the full prompt and both outputs without worrying about token count.

test_prompts = [
    "Explain model pruning in one sentence.",
    "What is the capital of France?",
    "Write a Python function to reverse a list.",
]

def generate_answer(target_model, prompt, max_new_tokens=60):
    inputs = tokenizer(prompt, return_tensors="pt")
    with torch.no_grad():
        outputs = target_model.generate(
            **inputs,
            max_new_tokens=max_new_tokens,
            do_sample=False,
            pad_token_id=tokenizer.eos_token_id,
        )
    return tokenizer.decode(outputs[0], skip_special_tokens=True)

original_outputs = [generate_answer(model, p) for p in test_prompts]
pruned_outputs = [generate_answer(pruned_model, p) for p in test_prompts]

results = []
for prompt, orig, pruned in zip(test_prompts, original_outputs, pruned_outputs):
    judge_input = (
        f"User prompt: {prompt}\n\n"
        f"Original output: {orig}\n\n"
        f"Pruned output: {pruned}\n\n"
        "Score the pruned output on coherence and factual consistency relative to the original. "
        "Respond with JSON: {\"score\": int, \"reason\": str}"
    )

    response = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=[
            {"role": "system", "content": "You are an expert evaluator of language model outputs."},
            {"role": "user", "content": judge_input},
        ],
    )

    verdict = json.loads(response.choices[0].message.content)
    results.append({"prompt": prompt, "score": verdict["score"], "reason": verdict["reason"]})
    print(f"Prompt: {prompt[:40]}... Score: {verdict['score']}/10")

Step 5: Aggregate scores and decide whether to keep the prune

I average the coherence scores and print a recommendation. If the average is above 8, I save the pruned weights. Otherwise, I tighten the sparsity target and rerun.

avg_score = sum(r["score"] for r in results) / len(results)
print(f"\nAverage quality score: {avg_score:.1f}/10")

if avg_score >= 8.0:
    print("Pruning is viable. Save the weights with torch.save(pruned_model.state_dict(), 'pruned_model.pt').")
else:
    print("Quality dropped too much. Reduce the sparsity ratio or prune fewer layers.")

Run it

Copy the blocks above into a single file named prune_agent.py, replace YOUR_OXLO_API_KEY, and run python prune_agent.py. Here is what the output looks like on my machine.

$ python prune_agent.py
Layers selected for pruning: [24, 26, 27, 28, 29]
Pruned 5 layers at 30% sparsity.
Prompt: Explain model pruning in one sentence... Score: 9/10
Prompt: What is the capital of France?... Score: 10/10
Prompt: Write a Python function to reverse a list... Score: 8/10

Average quality score: 9.0/10
Pruning is viable. Save the weights with torch.save(pruned_model.state_dict(), 'pruned_model.pt').

Wrap-up and next steps

This agent gives me a repeatable way to test pruning hypotheses without burning tokens on long evaluation prompts. Two concrete next steps: wire the judge loop into a CI test that fails if the score drops below a threshold, and try the same workflow on larger checkpoints by offloading reference generation to deepseek-v3.2 on Oxlo.ai while your laptop handles the sparse forward pass.

Using LLMs for Text Classification

shashank ms — Mon, 06 Jul 2026 19:37:25 +0000

What we are building

We are building a zero-shot support ticket classifier that routes incoming text into predefined categories without any custom model training. This is ideal for teams that need to triage large volumes of user feedback, and because we will run it on Oxlo.ai, long tickets with stack traces or logs cost the same flat rate as short ones.

What you'll need

Python 3.10 or newer
The OpenAI SDK: pip install openai
An Oxlo.ai API key from https://portal.oxlo.ai

Step 1: Set up the client and test a single classification

First, I will configure the OpenAI SDK to point at Oxlo.ai and send one ticket through Llama 3.3 70B to verify the setup and see the raw output.

from openai import OpenAI

client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")

SYSTEM_PROMPT = """You are a support ticket classifier.
Classify the ticket into exactly one category:
- Billing
- Technical Issue
- Account Access
- Feature Request
- General Inquiry

Respond with only the category name, nothing else."""

ticket = "I was charged twice for my subscription this month. Please refund the extra payment."

response = client.chat.completions.create(
    model="llama-3.3-70b",
    messages=[
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user", "content": ticket},
    ],
)

print(response.choices[0].message.content)

Step 2: Lock down categories in the system prompt

Hard-coding categories inside the prompt string gets messy fast. I will pull them into a Python list so I can reuse them in validation logic later, and keep the prompt itself clean.

CATEGORIES = [
    "Billing",
    "Technical Issue",
    "Account Access",
    "Feature Request",
    "General Inquiry",
]

SYSTEM_PROMPT = (
    "You are a support ticket classifier.\n"
    "Classify the ticket into exactly one category:\n"
    + "\n".join(f"- {c}" for c in CATEGORIES)
    + "\n\nRespond with only the category name, nothing else."
)

Step 3: Wrap the call in a classifier function

Now I will wrap the API call in a function that validates the model output against the allowed categories and falls back to "General Inquiry" if the model returns an unexpected string.

from openai import OpenAI

client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")

CATEGORIES = [
    "Billing",
    "Technical Issue",
    "Account Access",
    "Feature Request",
    "General Inquiry",
]

SYSTEM_PROMPT = (
    "You are a support ticket classifier.\n"
    "Classify the ticket into exactly one category:\n"
    + "\n".join(f"- {c}" for c in CATEGORIES)
    + "\n\nRespond with only the category name, nothing else."
)

def classify_ticket(text: str) -> str:
    response = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": text},
        ],
        temperature=0.0,
        max_tokens=20,
    )

    raw = response.choices[0].message.content.strip()

    for cat in CATEGORIES:
        if raw.lower() == cat.lower():
            return cat

    return "General Inquiry"

# Quick test
print(classify_ticket("How do I reset my password? I forgot it and the reset email is not arriving."))

Step 4: Classify a batch of tickets

In production you will handle many tickets at once. I will loop over a list, classify each one, and print a simple report. Because Oxlo.ai uses flat per-request pricing (https://oxlo.ai/pricing), a ticket with a long stack trace costs the same as a short question.

from openai import OpenAI

client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")

CATEGORIES = [
    "Billing",
    "Technical Issue",
    "Account Access",
    "Feature Request",
    "General Inquiry",
]

SYSTEM_PROMPT = (
    "You are a support ticket classifier.\n"
    "Classify the ticket into exactly one category:\n"
    + "\n".join(f"- {c}" for c in CATEGORIES)
    + "\n\nRespond with only the category name, nothing else."
)

def classify_ticket(text: str) -> str:
    response = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": text},
        ],
        temperature=0.0,
        max_tokens=20,
    )

    raw = response.choices[0].message.content.strip()

    for cat in CATEGORIES:
        if raw.lower() == cat.lower():
            return cat

    return "General Inquiry"

tickets = [
    "I was charged twice for my subscription this month. Please refund the extra payment.",
    "How do I reset my password? I forgot it and the reset email is not arriving.",
    "The API returns a 500 error when I send a request with an empty payload.",
    "It would be great if you added dark mode to the dashboard.",
    "Hi, I am wondering what your enterprise pricing looks like for a team of 50?",
]

for ticket in tickets:
    label = classify_ticket(ticket)
    print(f"{label:<20} | {ticket[:50]}...")

Run it

Save the full script from Step 4 as classify.py, replace YOUR_OXLO_API_KEY, and run it.

python classify.py

Example output:

Billing              | I was charged twice for my subscription this ...
Account Access       | How do I reset my password? I forgot it and th...
Technical Issue      | The API returns a 500 error when I send a requ...
Feature Request      | It would be great if you added dark mode to the...
General Inquiry      | Hi, I am wondering what your enterprise pricing...

Next steps

To make this production-ready, add a confidence threshold by asking the model to return a score from 1 to 10 and route low-confidence tickets to a human reviewer. You could also swap in qwen-3-32b or kimi-k2.6 on Oxlo.ai to compare latency and accuracy for your specific dataset without changing any other code.

Revolutionizing Accessibility with LLMs

shashank ms — Mon, 06 Jul 2026 13:34:33 +0000

Accessibility engineering demands processing inputs that are inherently verbose and unstructured. A screen reader capturing the full DOM tree of a complex web application, a real-time captioning pipeline buffering minutes of audio context, or an assistive agent maintaining a multi-turn conversation history all generate prompts that exceed typical consumer chat lengths. For developers building these tools, token-based inference pricing creates a direct conflict between comprehensiveness and cost. The longer the context required to serve the user, the steeper the bill, which forces teams to truncate valuable information or pass costs to end users.

The Long-Context Burden in Assistive Tech

Assistive technologies rarely operate on isolated sentences. They consume extended documents, persistent environmental descriptions, and lengthy user interaction logs. A text-to-speech tool summarizing a research paper for a visually impaired user might need to ingest the entire document to maintain coherence. An AI companion for neurodivergent users might reference hours of prior conversation to preserve context and tone. Under token-based billing, these necessary long-context operations become economically unpredictable. Developers are forced to truncate valuable context or pass costs to end users, undermining the social impact of the tool.

Predictable Pricing for Assistive Workloads

Oxlo.ai is a developer-first AI inference platform built on request-based pricing. Each API call incurs one flat cost per request regardless of prompt length. Unlike token-based providers such as Together AI, Fireworks AI, OpenRouter, Replicate, and Anyscale, cost does not scale with input length. For accessibility applications that depend on long-context comprehension and agentic workloads, Oxlo.ai is significantly cheaper because a 50,000-token prompt costs the same as a 500-token prompt. This predictability allows assistive tech teams to budget accurately and prioritize user needs over token economy. Request-based pricing can be 10x to 100x cheaper than token-based alternatives for long-context workloads, a difference that determines whether an assistive feature ships or stays experimental.

Multimodal Pipelines for Sensory Augmentation

Oxlo.ai hosts 45+ open-source and proprietary models across 7 categories, many directly applicable to accessibility pipelines. The Vision category includes Gemma 3 27B and Kimi VL A3B for image description and visual question answering, enabling real-time scene narration for blind and low-vision users. The Audio category offers Whisper Large v3, Whisper Turbo, and Whisper Medium for high-accuracy speech-to-text, alongside Kokoro 82M text-to-speech for natural voice synthesis. For developers building semantic retrieval over accessibility documentation or support libraries, the Embeddings category includes BGE-Large and E5-Large. All models are accessible through fully OpenAI API compatible endpoints, including chat/completions, audio/transcriptions, and audio/speech.

Building a Context-Aware Assistive Agent

Because Oxlo.ai is fully OpenAI SDK compatible, integration requires only a base URL change. The following Python example demonstrates how to stream a long-context request to Kimi K2.6, a model with advanced reasoning, agentic coding, vision capabilities, and a 131K context window. In this scenario, an assistive agent processes a lengthy screen-reader buffer and user instruction to generate a structured navigation plan.

from openai import OpenAI

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key="your-oxlo.ai-api-key"
)

# Simulated long-context input: extensive screen reader buffer + user goal
screen_buffer = """[Heading] Quarterly Report
[Link] Download CSV
[Table] Revenue by Region..."""  # Extends to thousands of tokens

response = client.chat.completions.create(
    model="kimi-k2-6",
    messages=[
        {"role": "system", "content": "You are an accessibility assistant. Convert screen reader buffers into concise, actionable navigation plans."},
        {"role": "user", "content": f"Screen buffer:\n{screen_buffer}\n\nGoal: Find the download link for the Q3 data."}
    ],
    stream=True,
    max_tokens=4096
)

for chunk in response:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="")

Because Oxlo.ai charges per request, sending the entire buffer in a single call does not trigger the exponential cost growth associated with token-based billing. The platform also supports streaming responses, function calling, JSON mode, vision input, and multi-turn conversations, making it suitable for interactive assistive interfaces.

Model Selection for Accessibility

Different accessibility tasks map to specific model strengths. For applications requiring deep reasoning across entire documents or extended conversation histories, DeepSeek V4 Flash offers an efficient MoE architecture, a 1 million token context window, and near state-of-the-art open-source reasoning. GLM 5, a 744B parameter MoE, targets long-horizon agentic tasks where an assistive agent must execute multi-step plans. For general-purpose assistance, Llama 3.3 70B and Qwen 3 32B provide robust multilingual reasoning. For coding-specific accessibility tools, Qwen 3 Coder 30B and DeepSeek Coder are available. Developers can prototype against 16+ free models on the Free tier before scaling to production.

Drop-In SDK Compatibility

Accessibility tools cannot tolerate cold starts. A screen reader waiting several seconds for model initialization breaks user trust. Oxlo.ai offers no cold starts on popular models, ensuring consistent latency for time-sensitive assistive interactions. The platform is a fully OpenAI SDK drop-in replacement, supporting Python, Node.js, and cURL. Existing accessibility projects built on OpenAI-compatible stacks can migrate to Oxlo.ai by updating the base URL to https://api.oxlo.ai/v1, with no refactoring of request logic.

Prototyping and Production

Oxlo.ai provides a Free plan at $0 per month with 60 requests per day, 16+ free models, and a 7-day full-access trial, which is ideal for accessibility hackers and nonprofit developers validating concepts. The Pro plan at $80 per month includes 1,000 requests per day across all models, while Premium at $350 per month offers 5,000 requests per day with priority queue access. For large-scale assistive deployments, Enterprise plans provide custom unlimited request volumes and dedicated GPUs, with guaranteed 30 percent savings versus your current provider. Exact request costs are detailed at https://oxlo.ai/pricing.

Conclusion

Building accessible AI means removing economic and technical barriers to long-context, multimodal inference. Oxlo.ai provides the model diversity, request-based pricing, and OpenAI-compatible infrastructure that accessibility developers need to ship features without compromising on context length or sensory modality.

Unlocking Creative Potential with LLMs

shashank ms — Mon, 06 Jul 2026 11:38:07 +0000

We are going to build a command-line creative writing partner that turns a one-sentence premise into a structured short story, complete with a title and back-cover blurb. It is useful for authors who want to blast through writer's block or content teams that need fictional prototypes fast. Because Oxlo.ai charges one flat rate per request, you can iterate on long outlines and full drafts without token costs scaling with every paragraph. See https://oxlo.ai/pricing for details.

What you'll need

Python 3.10 or newer
The OpenAI SDK: pip install openai
An Oxlo.ai API key from https://portal.oxlo.ai

Step 1: Set up the client and system prompt

I define a single system prompt that grounds the model in narrative craft. Keeping one prompt lets me reuse it across every stage of the pipeline.

SYSTEM_PROMPT = """You are an experienced fiction editor who specializes in tight, literary short fiction. You write in a modern, accessible prose style. You avoid clichés and deus ex machina endings. When asked for an outline, produce a three-act structure with one paragraph per act. When asked for prose, write complete scenes with sensory detail and dialogue. When asked for metadata, return compact JSON containing exactly two keys: title and blurb."""

Next I initialize the Oxlo.ai client. The OpenAI SDK works as a drop-in replacement, so only the base URL and API key change.

from openai import OpenAI

client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")

Step 2: Generate a three-act outline

The first function takes a user premise and returns a structured outline. I use Llama 3.3 70B because it handles general-purpose creative reasoning reliably.

def generate_outline(premise: str) -> str:
    user_message = (
        f"Write a three-act outline for a 1,000-word short story based on this premise: {premise}"
    )

    response = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": user_message},
        ],
    )

    return response.choices[0].message.content

Step 3: Draft the story

With the outline in hand, the second function asks the model for the full prose. I switch to Qwen 3 32B here to show how easily you can swap models on Oxlo.ai to match the task.

def draft_story(outline: str) -> str:
    user_message = (
        f"Turn the following outline into a complete 1,000-word short story.\n\n{outline}"
    )

    response = client.chat.completions.create(
        model="qwen-3-32b",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": user_message},
        ],
    )

    return response.choices[0].message.content

Step 4: Generate title and blurb

Finally, I extract marketing metadata. I ask for JSON so I can parse the result downstream. DeepSeek V3.2 handles structured instructions well, and Oxlo.ai supports JSON mode without extra configuration.

import json

def generate_metadata(story_text: str) -> dict:
    user_message = (
        f"Return JSON with a title and a one-sentence blurb for this story:\n\n{story_text}"
    )

    response = client.chat.completions.create(
        model="deepseek-v3.2",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": user_message},
        ],
        response_format={"type": "json_object"},
    )

    raw = response.choices[0].message.content
    return json.loads(raw)

Step 5: Assemble the pipeline

Now I wire the three functions into a single script. I print each stage so I can watch the creative pipeline unfold.

def main():
    premise = input("Story premise: ")

    print("\nGenerating outline...")
    outline = generate_outline(premise)
    print("\n--- OUTLINE ---\n", outline)

    print("\nDrafting story...")
    story = draft_story(outline)
    print("\n--- STORY ---\n", story)

    print("\nGenerating metadata...")
    meta = generate_metadata(story)
    print("\n--- METADATA ---\n", json.dumps(meta, indent=2))

if __name__ == "__main__":
    main()

Run it

Save everything into a file named story_builder.py, export your key, and run the script.

export OXLO_API_KEY="YOUR_OXLO_API_KEY"
python story_builder.py

Example session using the premise "A librarian discovers that overdue books predict the future":

Story premise: A librarian discovers that overdue books predict the future

Generating outline...

--- OUTLINE ---
Act I: Elena notices that every book returned late by a specific patron contains highlighted passages describing accidents that happen the next day...
Act II: She tests the pattern by withholding a return, and a predicted fire breaks out in the cafe across the street...
Act III: She confronts the patron, only to learn he is not predicting the future but rewriting it, and the overdue fees are the price of his guilt...

Drafting story...

--- STORY ---
The dust motes danced in the afternoon sun as Elena opened the returns bin. She had worked at the West Branch library for eleven years, and she knew every patron's habits...

Generating metadata...

--- METADATA ---
{
  "title": "The Late Prophecies",
  "blurb": "When a small-town librarian notices that overdue books describe tomorrow's disasters, she uncovers a patron who rewrites reality one page at a time."
}

Wrap-up and next steps

This pipeline gives you a reproducible way to go from idea to draft in under a minute. Because Oxlo.ai pricing is flat per request, you can rerun the pipeline or swap in larger context windows for novella-length outlines without worrying about token costs.

Two concrete next steps: add a revision loop where you feed the story back to the model with a critique prompt and ask for a rewritten second draft, or expose the pipeline through a lightweight FastAPI endpoint so a frontend can call it.

Multimodal LLMs for Vision-Language Tasks

shashank ms — Mon, 06 Jul 2026 11:35:52 +0000

Vision-language models have moved from research curiosities to production infrastructure. By combining a vision encoder with a large language model backbone, these systems can answer questions about images, parse documents, and coordinate agentic workflows that reason over visual inputs. For developers, the challenge is no longer proving the concept, but selecting the right model and inference backend for latency, context length, and cost structure.

What Are Vision-Language Models?

A vision-language model typically pairs a pretrained image encoder with an adapter or projection layer that translates visual features into the embedding space of a text-centric LLM. The resulting network consumes interleaved text and image tokens through a unified context window. Modern implementations treat images as sequences of patch embeddings, allowing the model to attend to spatial details with the same self-attention mechanisms used for text. This unification means function calling, JSON mode, and system prompts work identically across textual and multimodal inputs.

Architecture Patterns

Most production vision-language models follow one of two patterns. In the merged embedding architecture, a lightweight projector maps vision encoder outputs into LLM token embeddings before the first transformer layer. In the cross-attention architecture, visual features are injected into intermediate layers through dedicated attention modules. Merged embedding designs are simpler to serve and fine-tune, while cross-attention variants can preserve higher fidelity for dense visual reasoning. Both approaches rely on high-quality alignment data, usually interleaved image-text corpora or structured document understanding datasets.

Tasks That Benefit from Multimodal Reasoning

Multimodal LLMs excel at visual question answering, OCR, chart and diagram parsing, and UI element grounding. They also power agentic loops where a model observes a screenshot, plans the next action, and issues tool calls. Long-context windows are critical here. A single high-resolution image can consume thousands of tokens, and agentic traces that include multiple screenshots or video frames quickly exhaust short context limits. Providers that charge per token therefore impose a direct tax on visual complexity.

Implementation with Oxlo.ai

Oxlo.ai hosts several vision-capable models, including Gemma 3 27B, Kimi VL A3B, and Kimi K2.6. The platform is fully OpenAI SDK compatible, so switching from text-only to vision workloads requires only a model name change and an image payload.

Here is a minimal Python example that sends a base64-encoded image to the chat completions endpoint:

import base64
from openai import OpenAI

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key="YOUR_OXLO_API_KEY"
)

def encode_image(path):
    with open(path, "rb") as f:
        return base64.b64encode(f.read()).decode("utf-8")

image_b64 = encode_image("architecture.png")

response = client.chat.completions.create(
    model="gemma-3-27b-it",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "Explain the architecture in this diagram and list three potential bottlenecks."
                },
                {
                    "type": "image_url",
                    "image_url": {"url": f"data:image/png;base64,{image_b64}"}
                }
            ]
        }
    ],
    stream=False
)

print(response.choices[0].message.content)

You can enable streaming by setting stream=True, or enforce structured outputs with response_format={"type": "json_object"}. Because Oxlo.ai exposes the standard /v1/chat/completions schema, existing agent frameworks and eval pipelines drop in without modification.

Cost and Context Considerations

Images increase input length dramatically. On token-based providers, a single high-resolution image can translate into thousands of tokens, and agentic traces with multiple frames scale linearly. Oxlo.ai uses flat per-request pricing, so the cost of a vision call does not scale with input token count. For long-context and agentic workloads that process high-resolution images or extended video sequences, this model can be significantly cheaper than token-based alternatives. See the Oxlo.ai pricing page for plan details.

Selecting a Model on Oxlo.ai

Not every vision task requires the same capacity. Gemma 3 27B offers strong general vision-language performance and low latency for interactive applications. Kimi VL A3B provides efficient vision-language reasoning for document parsing and UI understanding. For the most demanding agentic coding or multistep visual reasoning tasks, Kimi K2.6 delivers advanced chain-of-thought reasoning, vision support, and a 131K context window. All models are served with no cold starts, so latency is predictable from the first request.

Conclusion

Vision-language capabilities are now a baseline expectation for production LLM stacks. The engineering decision comes down to context capacity, model selection, and inference economics. Oxlo.ai offers an OpenAI-compatible API, a range of vision-capable models from efficient to state-of-the-art, and a request-based pricing model that insulates multimodal workloads from the cost volatility of long image token sequences. If you are building agents or applications that see as well as reason, Oxlo.ai is a relevant option worth evaluating.

Building Dialogue Systems and Chatbots with LLMs

shashank ms — Mon, 06 Jul 2026 09:36:47 +0000

Building production dialogue systems with large language models requires more than prompt engineering. A robust chatbot architecture must manage conversation state, handle function calling, retrieve external knowledge, and serve responses under strict latency constraints. The infrastructure you choose determines whether these capabilities remain manageable at scale or become cost-prohibitive as context lengths grow.

Architecture of Modern Dialogue Systems

Traditional chatbot pipelines separated natural language understanding, dialogue management, and response generation into discrete components. Modern LLMs collapse these layers into a single inference call, but the surrounding architecture still needs deliberate design. A typical system includes a conversation buffer to serialize history, a router to classify intent and select tools, and a retrieval layer to inject relevant documents into the prompt.

When selecting a backend, compatibility matters. Oxlo.ai offers fully OpenAI SDK compatible endpoints, so existing chatbot codebases require only a base URL change to route requests to https://api.oxlo.ai/v1. This drop-in replacement preserves your investment in client libraries while giving you access to over 45 models spanning general reasoning, coding, vision, and embeddings.

Designing for Stateful Conversations

Dialogue quality depends on how well the system maintains context across turns. Long-running support sessions or agentic workflows can accumulate thousands of tokens of history. Under token-based pricing, every additional line of context increases cost linearly. Oxlo.ai uses request-based pricing, so a single API call costs the same flat amount regardless of prompt length. For chatbots that pass full conversation history or large system instructions on every turn, this structure eliminates the penalty for long-context messages.

Below is a minimal Python example using the OpenAI SDK with Oxlo.ai to maintain a multi-turn conversation:

from openai import OpenAI

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key="YOUR_OXLO_API_KEY"
)

messages = [
    {"role": "system", "content": "You are a technical support assistant. Be concise."},
    {"role": "user", "content": "My database connection keeps timing out."}
]

response = client.chat.completions.create(
    model="llama-3.3-70b",
    messages=messages
)

# Append assistant reply and continue
messages.append({"role": "assistant", "content": response.choices[0].message.content})

Extending Bots with Function Calling

Useful chatbots rarely operate in isolation. They query calendars, update tickets, or check inventory. Function calling lets an LLM emit structured JSON that triggers external tools. Oxlo.ai supports function calling and tool use across its chat models, including Llama 3.3 70B, Qwen 3 32B, and DeepSeek V3.2.

tools = [
    {
        "type": "function",
        "function": {
            "name": "check_ticket_status",
            "description": "Get the status of a support ticket",
            "parameters": {
                "type": "object",
                "properties": {
                    "ticket_id": {"type": "string"}
                },
                "required": ["ticket_id"]
            }
        }
    }
]

response = client.chat.completions.create(
    model="qwen-3-32b",
    messages=messages,
    tools=tools,
    tool_choice="auto"
)

if response.choices[0].message.tool_calls:
    # Route to your handler
    pass

Memory and Retrieval

Once a conversation exceeds the context window, or when a user returns after a session ends, you need external memory. The standard pattern stores conversation summaries or user facts in a vector database, then retrieves them via embedding search. Oxlo.ai provides dedicated embedding endpoints through models like BGE-Large and E5-Large, letting you generate vectors without managing a separate provider. You can keep the entire pipeline, from retrieval to generation, on a single platform with unified billing.

Latency and Reliability in Production

Users expect chatbots to stream tokens as they are generated, not to wait for an entire response to materialize. Oxlo.ai supports streaming responses across its chat models, with no cold starts on popular models. That consistency matters when you are building interactive experiences where a perceptible delay breaks immersion.

Cost Engineering at Scale

Token-based providers such as Together AI, Fireworks AI, OpenRouter, Replicate, and Anyscale scale cost linearly with prompt length. That penalizes the exact behaviors that make chatbots effective: lengthy system prompts, few-shot examples, full conversation history, and agentic loops that iterate over tools. Because Oxlo.ai charges one flat cost per request, long-context and agentic workloads can be significantly cheaper than token-based alternatives. You can view the exact structure on the Oxlo.ai pricing page.

When architecting a dialogue system, estimate your average context length per turn. If your prompts routinely exceed a few thousand tokens, request-based pricing removes the scaling tax on input size and makes cost forecasting straightforward.

Conclusion

Building dialogue systems with LLMs is an infrastructure problem as much as a modeling one. You need stateful context, reliable tool use, external memory, and predictable economics. Oxlo.ai provides the model variety, OpenAI compatible SDK, and request-based pricing that align with production chatbot requirements. If you are evaluating backends for your next conversational AI project, route a test conversation to https://api.oxlo.ai/v1 and measure the difference in both latency and cost structure.

LLMs for Text Classification and Sentiment Analysis: A Comprehensive Guide

shashank ms — Mon, 06 Jul 2026 09:34:20 +0000

We are going to build a production-ready text classifier that reads customer support tickets, assigns a category, scores sentiment, and flags urgency. It runs entirely on Oxlo.ai's request-based API, so long tickets cost the same flat rate as short ones. This is ideal for teams that need to process large backlogs or threaded conversations without worrying about token costs.

What you'll need

Python 3.10 or newer
An Oxlo.ai API key from https://portal.oxlo.ai
The OpenAI SDK: pip install openai

Step 1: Set up the Oxlo.ai client

I start by initializing the OpenAI SDK to point at Oxlo.ai. I use Llama 3.3 70B as the default classifier because it follows instructions tightly and keeps output concise.

from openai import OpenAI
import json
import os

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key=os.environ.get("OXLO_API_KEY", "YOUR_OXLO_API_KEY")
)

CLASSIFIER_MODEL = "llama-3.3-70b"

Step 2: Write the system prompt

The prompt enforces a strict JSON schema. I also ask for a confidence score so I can route low-confidence predictions to a stronger model later.

SYSTEM_PROMPT = """You are a support-ticket classifier. Analyze the user message and return a JSON object with exactly these keys:
- category: one of ["Billing", "Technical", "Feature Request", "Other"]
- sentiment: one of ["Positive", "Neutral", "Negative"]
- urgency: one of ["Low", "Medium", "High", "Critical"]
- confidence: a float between 0.0 and 1.0 representing your certainty
- reasoning: a single sentence explaining your choice

Rules:
- Return only the JSON object, with no markdown formatting.
- If the user mentions downtime or data loss, set urgency to Critical.
- If the user says thank you or praises the product, sentiment is Positive."""

Step 3: Build the core classifier

This function sends the ticket to Oxlo.ai with JSON mode enabled. The temperature is set low to keep outputs deterministic.

def classify_ticket(text: str) -> dict:
    response = client.chat.completions.create(
        model=CLASSIFIER_MODEL,
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": text},
        ],
        response_format={"type": "json_object"},
        temperature=0.1,
        max_tokens=256,
    )
    
    content = response.choices[0].message.content
    return json.loads(content)

Step 4: Process batches safely

Real pipelines handle many tickets at once. I wrap the single-ticket call in a loop that catches parsing errors and marks failed rows so one bad response does not crash the job.

from typing import List

def classify_batch(tickets: List[str]) -> List[dict]:
    results = []
    for text in tickets:
        try:
            result = classify_ticket(text)
            result["input_preview"] = text[:120]
            results.append(result)
        except Exception as e:
            results.append({
                "input_preview": text[:120],
                "error": str(e),
                "category": "Unknown",
                "sentiment": "Neutral",
                "urgency": "Low",
                "confidence": 0.0,
                "reasoning": "Parsing failed"
            })
    return results

Step 5: Add confidence-based fallback

If the fast model is unsure, I send the ticket to DeepSeek V3.2, which excels at reasoning and coding contexts. Because Oxlo.ai pricing is per request, the fallback costs the same flat rate as the first attempt, even if the ticket contains a long stack trace or thread history.

STRONG_MODEL = "deepseek-v3.2"

def classify_with_fallback(text: str, threshold: float = 0.7) -> dict:
    first = classify_ticket(text)
    
    if first.get("confidence", 1.0) < threshold:
        response = client.chat.completions.create(
            model=STRONG_MODEL,
            messages=[
                {"role": "system", "content": SYSTEM_PROMPT},
                {"role": "user", "content": text},
            ],
            response_format={"type": "json_object"},
            temperature=0.1,
            max_tokens=256,
        )
        second = json.loads(response.choices[0].message.content)
        second["model_used"] = STRONG_MODEL
        return second
    
    first["model_used"] = CLASSIFIER_MODEL
    return first

Step 6: Generate a summary report

I add a small reporter so I can scan the output without reading raw JSON.

def print_report(results: List[dict]):
    print(f"Processed {len(results)} tickets\n")
    for r in results:
        tag = r.get("urgency", "Low")
        print(f"[{tag}] {r.get('category', '?'):15} | "
              f"sentiment={r.get('sentiment', '?'):8} | "
              f"confidence={r.get('confidence', 0):.2f} | "
              f"model={r.get('model_used', '?')}")
        print(f"    reasoning: {r.get('reasoning', 'N/A')}\n")

Run it

Here is a small test set. I run each ticket through the fallback classifier and print the report.

if __name__ == "__main__":
    tickets = [
        "I was charged twice last month and I need a refund immediately.",
        "Love the new dashboard, great work team!",
        "The API returns a 500 error every time I try to export my data.",
        "Can you add dark mode to the mobile app? It would really help at night.",
    ]

    classified = [classify_with_fallback(t) for t in tickets]
    print_report(classified)

Example output:

Processed 4 tickets

[High] Billing         | sentiment=Negative | confidence=0.92 | model=llama-3.3-70b
    reasoning: User reports incorrect double charge and requests refund.

[Low]  Feature Request | sentiment=Positive | confidence=0.88 | model=llama-3.3-70b
    reasoning: User praises product and suggests new feature.

[Critical] Technical   | sentiment=Negative | confidence=0.95 | model=llama-3.3-70b
    reasoning: User reports repeated server errors affecting data export.

[Low]  Feature Request | sentiment=Neutral  | confidence=0.76 | model=llama-3.3-70b
    reasoning: User asks for dark mode without expressing strong emotion.

Wrap-up

Two concrete next steps. First, wire the classifier into a webhook so it runs on every new Zendesk or Intercom ticket and tags them automatically. Second, add a vector embedding step using Oxlo.ai's BGE-Large endpoint to cluster similar tickets and detect emerging issues before they become critical. You can explore request-based pricing for high-volume workloads at https://oxlo.ai/pricing.