DEV Community: shashank ms

Building Chatbots with LLM, NLU, and Speech Recognition

shashank ms — Wed, 09 Sep 2026 05:33:50 +0000

We will build a voice-first support triage bot that ingests customer audio, transcribes it with Whisper, extracts intent via structured LLM inference, and drafts a contextual reply. It is useful for SaaS teams that want to automate first-line support without gluing together separate providers for speech, NLU, and dialogue. Oxlo.ai hosts Whisper, Qwen, and Llama behind one OpenAI-compatible endpoint, so the entire pipeline runs against a single base URL.

What you'll need

Python 3.10 or newer
The OpenAI SDK: pip install openai
An Oxlo.ai API key from https://portal.oxlo.ai
A sample audio file named support_request.wav for testing

Step 1: Configure the client

Instantiate the SDK once and point it at Oxlo.ai. I pull the key from the environment so it never sits in source control.

import os
import json
from openai import OpenAI

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key=os.getenv("OXLO_API_KEY"),
)

Step 2: Transcribe speech

Send the audio to Oxlo.ai's transcription endpoint. I use whisper-large-v3 because it handles noisy microphone audio and accents well.

def transcribe(audio_path: str) -> str:
    with open(audio_path, "rb") as audio_file:
        result = client.audio.transcriptions.create(
            model="whisper-large-v3",
            file=audio_file,
        )
    return result.text

user_input = transcribe("support_request.wav")
print("Transcript:", user_input)

Step 3: Extract intent and entities

Feed the raw transcript to qwen-3-32b with a strict system prompt that forces JSON. I enable JSON mode so the output stays machine readable without regex hacks.

INTENT_PROMPT = """You are an NLU engine.
Read the support message and return a JSON object with exactly these keys:
- intent: one of [billing, technical, sales, unknown]
- product: the product mentioned, or null
- urgency: one of [low, medium, high]
- summary: a 10-word summary of the issue
Respond with valid JSON only. No markdown, no explanation."""

def extract_entities(transcript: str) -> dict:
    response = client.chat.completions.create(
        model="qwen-3-32b",
        messages=[
            {"role": "system", "content": INTENT_PROMPT},
            {"role": "user", "content": transcript},
        ],
        response_format={"type": "json_object"},
    )
    return json.loads(response.choices[0].message.content)

entities = extract_entities(user_input)
print("Entities:", entities)

Step 4: Draft the agent response

Pass the original transcript and parsed entities to llama-3.3-70b. The system prompt below adapts tone based on urgency and intent, keeping replies short so they work well in voice or chat windows.

SYSTEM_PROMPT = """You are a concise technical support agent for a SaaS platform.
Use the provided intent and entity JSON to tailor your answer.
If urgency is high, acknowledge it and offer an escalation path.
If the intent is billing, keep the tone calm and direct.
If technical, ask one clarifying question maximum.
Always respond in two sentences or fewer."""

def draft_response(transcript: str, entities: dict) -> str:
    context = f"User said: {transcript}\nExtracted entities: {json.dumps(entities)}"
    response = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": context},
        ],
    )
    return response.choices[0].message.content

reply = draft_response(user_input, entities)
print("Agent:", reply)

Run it

Tie the stages together and execute the full pipeline against your sample audio.

if __name__ == "__main__":
    audio_file = "support_request.wav"

    transcript = transcribe(audio_file)
    entities = extract_entities(transcript)
    reply = draft_response(transcript, entities)

    print("\n--- Final Output ---")
    print(f"Intent:  {entities['intent']}")
    print(f"Urgency: {entities['urgency']}")
    print(f"Reply:   {reply}")

Expected output:

Transcript: My dashboard won't load since this morning and I have a demo in an hour.
Entities: {'intent': 'technical', 'product': 'dashboard', 'urgency': 'high', 'summary': 'Dashboard not loading before demo'}
Agent: I see the dashboard is not loading and your demo is soon. Have you tried a hard refresh or clearing your browser cache?

--- Final Output ---
Intent:  technical
Urgency: high
Reply:   I see the dashboard is not loading and your demo is soon. Have you tried a hard refresh or clearing your browser cache?

Next steps

Wrap the pipeline in a FastAPI endpoint so your frontend can stream user voice uploads directly to the bot. Alternatively, push the final reply through Oxlo.ai's Kokoro TTS endpoint to return spoken audio instead of text, giving you a true voice-in, voice-out agent.

Token-Based LLM API with Request-Based Pricing

shashank ms — Wed, 09 Sep 2026 05:32:08 +0000

Developers choosing an LLM API today face a hidden cost curve. Token-based billing, the default across most inference providers, charges for every input and output token. For long-context retrieval, agentic loops, or large codebases, this means costs scale linearly with prompt length. Request-based pricing flips the model: one flat cost per API call regardless of how many tokens you send. This article breaks down the architectural trade-offs, shows where flat pricing wins, and how to integrate it with Oxlo.ai.

The Token Pricing Problem

Token-based billing is straightforward in theory. You pay for what you consume. In practice, modern workloads rarely fit neat token budgets. A single agent turn might include a 128k system prompt, a retrieved document chunk, and multi-turn history. Under token pricing, that one request can cost as much as dozens of shorter calls. Providers like Together AI, Fireworks AI, OpenRouter, Replicate, and Anyscale all use token-centric models. As context windows grow and agents iterate in loops, token counts explode and budgets become unpredictable.

How Request-Based Pricing Changes the Math

Request-based pricing decouples cost from prompt size. You pay one flat fee per HTTP request, whether you send 500 tokens or 100,000. This makes budgeting deterministic. A developer running 1,000 requests per day knows the exact bill before the first request leaves the client. It also removes the penalty for rich context. You can include full documentation, lengthy chat histories, or large schema definitions without watching a meter tick upward. Oxlo.ai uses this model, offering a flat per-request rate across its entire catalog. For exact rates, see the Oxlo.ai pricing page.

When Flat Pricing Wins

Long-context workflows are the obvious fit. RAG pipelines that stuff retrieved passages into the prompt, code review agents that ingest entire repositories, and multi-turn customer support bots all generate large inputs. Under token pricing, these are premium workloads. Under request-based pricing, they are standard API calls.

Agentic systems amplify the benefit. An agent that calls tools, appends results, and re-prompts the model in a loop can accumulate tens of thousands of tokens per step. With request pricing, each loop iteration is a fixed cost, so you can design for accuracy instead of token economy.

Oxlo.ai specifically targets this profile. Its request-based structure can be 10-100x cheaper than token-based alternatives for long-context workloads, and it offers models like DeepSeek V4 Flash with 1M context windows and Kimi K2.6 with 131K context without per-token surcharges.

Integrating Oxlo.ai

Oxlo.ai is fully OpenAI SDK compatible. Switching from a token-based provider takes minutes. Change the base URL and API key, and existing code runs unchanged. There are no cold starts on popular models, so latency stays consistent.

from openai import OpenAI

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key="your-oxlo.ai-api-key"
)

response = client.chat.completions.create(
    model="deepseek-v4-flash",
    messages=[
        {"role": "system", "content": "You are a coding assistant."},
        {"role": "user", "content": "Refactor this large codebase..."}  # 100k+ tokens
    ],
    stream=True
)

for chunk in response:
    print(chunk.choices[0].delta.content, end="")

Because Oxlo.ai bills per request, the size of the refactor prompt does not change the cost. Streaming, function calling, JSON mode, and vision are all supported through the same endpoints.

Model Availability

A request-based platform is only useful if it carries the models you need. Oxlo.ai hosts 45+ open-source and proprietary models across seven categories.

For reasoning and chat, you have Qwen 3 32B, Llama 3.3 70B, DeepSeek R1 671B MoE, GPT-Oss 120B, Kimi K2.6, GLM 5, and Minimax M2.5. For coding, there is Qwen 3 Coder 30B, DeepSeek Coder, and Oxlo.ai Coder Fast. Vision workloads can use Gemma 3 27B or Kimi VL A3B. Image generation, audio transcription, text-to-speech, embeddings, and object detection are also available under the same flat request model.

This breadth means you are not sacrificing model quality for pricing predictability.

Choosing Your Pricing Model

Token-based pricing works for sporadic, short-prompt workloads where usage is low and unpredictable. Request-based pricing wins when you run agents, process long documents, or simply want a flat operational budget.

Oxlo.ai offers a free tier with 60 requests per day across 16+ models, so you can test the pricing model against your actual workload without committing. Paid plans scale to thousands of requests per day with priority queues and dedicated GPU options for enterprise teams. For exact rates, see the Oxlo.ai pricing page.

The shift toward long-context and agentic AI is exposing the friction in token-based billing. Request-based pricing aligns costs with developer workflows, not character counts. If your prompts are growing and your token budget is not, Oxlo.ai is a relevant, drop-in alternative that keeps costs flat without restricting model choice.

Optimizing LLM Inference for Low Memory Usage and High Accuracy

shashank ms — Wed, 09 Sep 2026 03:35:43 +0000

Memory pressure is the primary bottleneck when deploying large language models at scale. As context lengths grow and agentic workflows multiply, the cost of loading weights and maintaining the KV cache often exceeds the cost of forward computation. This article covers practical techniques to minimize memory footprint without destroying accuracy, from quantization strategies to architectural choices. We will also look at how to offload infrastructure complexity so your team can focus on model behavior rather than GPU memory limits.

Quantization and Precision Formats

The simplest way to cut memory usage is to reduce the bit width of weights and activations. Modern formats go far beyond naive INT8 rounding. FP8 (E4M3 for weights, E5M2 for gradients) on NVIDIA Hopper GPUs retains near-baseline accuracy for inference while halving model size relative to FP16. For consumer GPUs or CPU offload, GGUF with Q4_K_M and Q5_K_M block quantization offers a strong accuracy-to-size ratio. Activation-aware methods like AWQ and GPTQ protect salient weight channels during 4-bit compression, which preserves reasoning performance better than uniform INT4.

The tradeoff is task-dependent. FP8 is usually safe for code generation and long-context retrieval. INT4 GGUF can degrade multi-step math reasoning, so benchmark your specific workload before deploying. If you are self-hosting, load the model with transformers or llama.cpp using the native format that matches your hardware. If you would rather skip quantization tuning entirely, Oxlo.ai serves optimized variants of models like DeepSeek R1 671B MoE, Qwen 3 32B, and Llama 3.3 70B with infrastructure-level memory management already applied.

KV Cache and Memory Management

For long sequences, the KV cache often consumes more GPU memory than the model weights themselves. Standard multi-head attention stores separate key and value tensors for every head, but grouped-query attention (GQA) and multi-query attention (MQA) cut cache size by sharing KV heads across query heads. Most modern open-weight models, including Llama 3.3 70B and Qwen 3, already use GQA, so prefer them over pure MHA architectures when memory is tight.

Further reductions come from KV cache quantization to INT8 and dynamic allocation strategies like PagedAttention. Instead of reserving contiguous blocks for the full maximum sequence length, PagedAttention allocates fixed-size pages on demand, eliminating internal fragmentation. When self-hosting with vLLM or TGI, enable prefix caching to reuse KV tensors across repeated system prompts or multi-turn conversations.

On the client side, keep contexts concise. Summarize earlier conversation turns, truncate irrelevant documents, and store embeddings for retrieval instead of stuffing full texts into the prompt. These habits reduce cache pressure regardless of your provider.

Attention and Context Window Optimization

Not every token in a 128K context needs full pairwise attention. Sliding window attention, used in models like Mistral, restricts the receptive field to local neighbors and a few global tokens. This lowers memory complexity from quadratic to near-linear in practice. FlashAttention-3 and scaled dot-product attention (SDPA) fused kernels also reduce high-bandwidth memory traffic by keeping attention computations in SRAM as long as possible.

If your task truly requires 1M tokens, choose an architecture designed for it. DeepSeek V4 Flash supports 1M context windows with an efficient MoE design, and Kimi K2.6 handles 131K contexts with advanced reasoning. Sending a 500K token prompt to a model that lacks sparse attention or efficient KV paging will cause out-of-memory errors or punitive latency.

When chunking long documents, overlap chunks by a few sentences and use an embedding model to route queries to the most relevant chunk. Oxlo.ai offers embedding endpoints like BGE-Large and E5-Large that work well for this preprocessing step.

Batching and Throughput

Static batching wastes memory because every request in the batch must pad to the longest sequence. Continuous batching, also called in-flight batching, dynamically replaces completed sequences with new ones at every forward pass. This keeps GPU utilization high and memory fragmentation low. If you self-host, enable this in vLLM or TensorRT-LLM.

However, batching introduces scheduling complexity. You must balance throughput against time-to-first-token (TTFT) and time-between-tokens (TBT). For applications with bursty traffic, maintaining a warm pool of GPU workers is expensive. Oxlo.ai eliminates this operational burden with no cold starts on popular models, so you can send requests individually and still benefit from server-side continuous batching.

Model Selection and Architecture

Mixture-of-Experts (MoE) models such as DeepSeek R1 671B and GLM 5 load all parameters into memory but activate only a subset per token. This can improve quality without proportional compute cost, though memory requirements for the full parameter set remain high. For constrained environments, smaller dense models like Qwen 3 32B or Llama 3.3 70B often provide better latency and simpler deployment.

Task-specific models also reduce bloat. Use Qwen 3 Coder 30B or Oxlo.ai Coder Fast for programming tasks instead of a general 70B model. For vision tasks, Gemma 3 27B or Kimi VL A3B are more efficient than piping images through a massive text-only LLM. Offloading transcription to Whisper Large v3 and image generation to Flux.1 or Oxlo.ai Image Pro keeps your LLM context free for reasoning.

Client-Side Optimizations

Even with a perfectly optimized backend, inefficient client code can spike memory and cost. Use streaming responses to begin processing output before generation finishes. Set explicit max_tokens and stop sequences to prevent runaway completions. Use JSON mode or constrained decoding when you need structured output, which shortens generation length and reduces cache lifetime.

Here is a minimal Python example using the OpenAI SDK with Oxlo.ai. It sets a token limit, enables streaming, and uses JSON mode to constrain the response format:

import openai

client = openai.OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key="your-api-key"
)

response = client.chat.completions.create(
    model="qwen-3-32b",
    messages=[
        {"role": "system", "content": "You are a concise technical assistant."},
        {"role": "user", "content": "Summarize the KV cache optimization techniques in 3 sentences."}
    ],
    max_tokens=150,
    response_format={"type": "json_object"},
    stop=["\n\n"],
    stream=True
)

for chunk in response:
    print(chunk.choices[0].delta.content or "", end="")

Because Oxlo.ai uses request-based pricing, the cost of this call is flat

Using LLM for Text Summarization and Question Answering

shashank ms — Wed, 09 Sep 2026 03:34:43 +0000

We are building a document assistant that reads long articles or reports, produces a structured summary, and answers follow-up questions using the original text as ground truth. This is useful for research teams, support engineers, or anyone who needs to extract signal from dense material quickly.

What you'll need

Python 3.10 or newer
An Oxlo.ai API key from https://portal.oxlo.ai
The OpenAI SDK: pip install openai

Step 1: Configure the Oxlo.ai client

Replace YOUR_OXLO_API_KEY with the key from your Oxlo.ai dashboard. I use llama-3.3-70b as the default because it handles both summarization and reasoning in one call. If you are processing books or legal briefs, you can swap in kimi-k2.6 later for its 131K context window.

from openai import OpenAI

client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")

MODEL = "llama-3.3-70b"

Step 2: Define the system prompt

The system prompt needs to be strict. I tell the model to ground every answer in the provided text, avoid speculation, and format summaries as bullet points. I store it as a constant so I can tweak it without touching business logic.

SYSTEM_PROMPT = """You are a precise document assistant. Your job has two parts:

1. Summarization: When the user provides a document under the tag <document>, output a structured summary with three sections: Key Points, Named Entities, and One-Sentence Takeaway.
2. Question Answering: When the user asks a question, answer using only the information inside the most recent <document> tags. If the answer is not in the text, say "The document does not specify."

Rules:
- Use bullet points for lists.
- Do not invent facts.
- Keep answers under 150 words unless the user asks for detail."""

Step 3: Summarize long documents

I wrap the raw text in XML-like tags so the model knows what is source material versus instruction. This reduces prompt injection and keeps the context boundary clean. Because Oxlo.ai uses request-based pricing, sending a 10K word article in a single call costs the same as a one-liner, which makes long-context summarization practical without token math.

def summarize_document(text: str) -> str:
    user_message = f"Please summarize the following document:\n\n<document>\n{text}\n</document>"
    
    response = client.chat.completions.create(
        model=MODEL,
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": user_message},
        ],
        temperature=0.3,
        max_tokens=1024,
    )
    return response.choices[0].message.content

Step 4: Answer questions from source

For QA, I append the question after the document block so the model has the full source in the same context window. This avoids retrieval errors and keeps the implementation stateless. If your documents routinely exceed the context limit, chunk them and run map-reduce, but for most reports a single call works fine.

def ask_question(text: str, question: str) -> str:
    user_message = (
        f"<document>\n{text}\n</document>\n\n"
        f"Question: {question}"
    )
    
    response = client.chat.completions.create(
        model=MODEL,
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": user_message},
        ],
        temperature=0.1,
        max_tokens=512,
    )
    return response.choices[0].message.content

Step 5: Wire the interactive loop

I tie the two functions together in a small CLI. It loads a sample document, prints the summary, then enters a loop where the user can ask questions. In production you would swap stdin for an API endpoint, but this version is enough to validate the behavior.

SAMPLE_DOC = """
Artificial intelligence adoption in enterprise software has accelerated since 2023.
Key drivers include API-based access to large language models, reduced infrastructure costs,
and improved retrieval pipelines. Security remains the primary blocker, with 60 percent of
CIOs citing data residency as a top concern. Vendors who offer self-hosted or VPC deployment
options are winning evaluations in regulated industries. The report predicts that by 2026,
over 50 percent of new enterprise applications will embed LLM-powered features natively.
"""

if __name__ == "__main__":
    print("Generating summary...\n")
    summary = summarize_document(SAMPLE_DOC)
    print(summary)
    print("\n---\n")

    while True:
        try:
            q = input("Ask a question (or type 'quit'): ").strip()
            if q.lower() in ("quit", "exit"):
                break
            answer = ask_question(SAMPLE_DOC, q)
            print(f"\nAnswer: {answer}\n")
        except KeyboardInterrupt:
            break

Run it

Save the script as doc_agent.py, set your YOUR_OXLO_API_KEY, then run python doc_agent.py. You should see output similar to this:

Generating summary...

Key Points:
- Enterprise AI adoption has accelerated since 2023 due to API access, lower costs, and better retrieval.
- Security and data residency are the main blockers for CIOs.
- Self-hosted or VPC options are winning in regulated industries.
- By 2026, over 50 percent of new enterprise apps will embed LLM features natively.

Named Entities:
- CIOs, regulated industries, enterprise software vendors.

One-Sentence Takeaway:
- Enterprise AI is growing rapidly, but security and deployment flexibility are deciding factors in procurement.

---

Ask a question (or type 'quit'): What do CIOs care about most?
Answer: The document states that 60 percent of CIOs cite data residency as a top concern.

Ask a question (or type 'quit'): Who wrote the report?
Answer: The document does not specify.

Wrap-up and next steps

Swap in kimi-k2.6 if you need to process full PDFs without chunking, since its 131K context window handles most books in one shot. If you want to expose this as a service, wrap the functions in FastAPI and stream responses using Oxlo.ai's streaming support. You can view request-based pricing details at https://oxlo.ai/pricing.

Deploying LLM Models on Mobile Devices with Low Power Consumption

shashank ms — Wed, 09 Sep 2026 03:33:36 +0000

The push to run large language models directly on phones and tablets is driven by three hard requirements: latency, privacy, and offline availability. But the physics of mobile hardware creates a ceiling. NPUs and DSPs on flagship SoCs are powerful, yet thermal design power and battery capacity turn long-context inference or multi-turn reasoning into a rapid drain. The practical path forward is not all-edge or all-cloud. It is a tiered architecture where small, quantized models handle sensitive, frequent tasks locally, and a predictable cloud API handles everything else.

Model Selection and Quantization for Mobile

To keep power draw under control, the model must fit into device DRAM without constant swapping, and the working set must be small enough to avoid sustained high-frequency memory clocks. For most current mobile hardware, this means targeting models between 1B and 4B parameters, quantized to INT4 or INT8.

Strong candidates include Llama 3.2 1B and 3B, Qwen 2.5 0.5B through 3B, Phi-3 Mini 3.8B, and Gemma 2B and 4B. These architectures use grouped-query attention or multi-query attention, which shrinks the KV cache and reduces memory bandwidth, one of the largest contributors to energy consumption on mobile SoCs.

Use quantization formats that your runtime supports natively. GGUF via llama.cpp is the most common path for rapid prototyping. For production Android apps, ONNX Runtime with INT8 QDQ graphs and Qualcomm QNN delegates lets you execute on the Hexagon NPU. On iOS, Core ML Tools converts models to use the Neural Engine.

from llama_cpp import Llama

llm = Llama(
    model_path="./qwen2.5-1.5b-q4_k_m.gguf",
    n_ctx=2048,
    n_threads=4,
    verbose=False
)
output = llm.create_chat_completion(
    messages=[{"role": "user", "content": "Summarize this paragraph."}]
)

Inference Engines and Runtime Targets

The choice of runtime determines whether you are burning watts on the CPU or executing efficiently on the NPU or GPU.

llama.cpp. The de facto standard for mobile LLM inference. It supports Apple Metal on iOS and ARM NEON / dotprod on Android. Vulkan GPU offload is available for Adreno and Mali GPUs. It is the easiest path for GGUF models.
MediaPipe LLM Inference API. A cross-platform Google solution that bundles model weights and handles memory mapping. It is useful if you want identical code across Android and iOS and do not need custom quantization.
ONNX Runtime Mobile. Best when you need delegate support for hardware accelerators. The Qualcomm Neural Network (QNN) delegate runs on Hexagon NPU, and the Core ML delegate targets Apple Neural Engine. This is the path to lowest sustained power for larger sub-4B models.</

Large Language Model Architectures with Chain-of-Thought Reasoning

shashank ms — Wed, 09 Sep 2026 01:36:36 +0000

Chain-of-thought reasoning has moved from a prompting technique to a core architectural requirement for large language models. Instead of compressing hidden reasoning into a single forward pass, modern training pipelines explicitly optimize models to emit intermediate reasoning steps before producing a final answer. This shift changes how developers should think about inference costs, context windows, and model selection, especially when reasoning traces grow to thousands of tokens.

From Prompting to Architecture

Early chain-of-thought work relied on few-shot prompting to coax reasoning out of generalist models. Today, architectures are trained with reinforcement learning on verifiable rewards or supervised fine-tuning on curated reasoning traces. Models like DeepSeek R1 671B MoE and Kimi K2 Thinking expose long internal monologues as part of their standard generation behavior. This is not a post-hoc add-on, but a fundamental change in how transformers allocate compute across layers and how attention heads represent intermediate logical states.

Mixture-of-Experts and Reasoning Efficiency

Reasoning workloads amplify the cost of dense attention. Mixture-of-Experts architectures mitigate this by activating only a subset of parameters per token. DeepSeek R1 671B MoE and GLM 5 744B MoE use sparse activation to deliver deep reasoning without provisioning every parameter on every forward pass. For developers, this means state-of-the-art chain-of-thought quality becomes feasible without dedicated hardware clusters. The trade-off is latency variability, but the quality gains on mathematical and coding benchmarks are substantial.

Long Context and Reasoning Traces

Chain-of-thought models can generate thousands of tokens of internal reasoning before answering. When these traces are fed back into multi-turn agentic loops, total prompt and completion length grows rapidly. On token-based billing platforms, long reasoning traces translate directly into unpredictable costs. Oxlo.ai uses request-based pricing, so a single API call costs one flat rate regardless of how many reasoning tokens the model emits. For long-context reasoning workloads, this can be 10-100x cheaper than token-based alternatives because cost does not scale with input or output length. That predictability makes Oxlo.ai a practical choice for agentic workflows and long-horizon reasoning tasks where trace length is unknown upfront.

Model Selection for Reasoning Workloads

Different reasoning architectures excel in different domains. Oxlo.ai hosts more than 45 models across seven categories, all behind a single OpenAI-compatible endpoint, so switching between reasoning architectures requires only a model name change.

DeepSeek V4 Flash offers a 1 million token context window and near state-of-the-art open-source reasoning, making it suitable for document analysis with step-by-step extraction.
Kimi K2.6 combines advanced reasoning with vision and a 131K context, which helps when reasoning over charts or diagrams.
Kimi K2.5 and Kimi K2 Thinking provide advanced chain-of-thought reasoning for logic-heavy tasks.
Qwen 3 32B delivers multilingual reasoning and agent workflows for global applications.
DeepSeek R1 671B MoE remains a flagship for complex coding and deep reasoning.
GPT-Oss 120B offers a large open-source alternative for general reasoning.
GLM 5 targets long-horizon agentic tasks with its 744B MoE architecture.

Developers can prototype chain-of-thought pipelines on the Oxlo.ai free tier, which includes DeepSeek V3.2 for coding and reasoning workloads at 60 requests per day.

Implementing Chain-of-Thought on Oxlo.ai

Because Oxlo.ai is fully OpenAI SDK compatible, you can invoke reasoning models with the same client code you already use. The following example calls DeepSeek R1 671B MoE to solve a logic puzzle with explicit step-by-step reasoning.

import openai

client = openai.OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key="YOUR_OXLO_API_KEY"
)

response = client.chat.completions.create(
    model="deepseek-r1-671b",
    messages=[
        {
            "role": "system",
            "content": "You are a precise reasoning engine. Show your chain of thought before giving the final answer."
        },
        {
            "role": "user",
            "content": "Three switches control three bulbs in another room. You can enter the room only once. How do you determine which switch controls which bulb?"
        }
    ],
    stream=False
)

print(response.choices[0].message.content)

With request-based pricing, the length of the reasoning trace does not affect cost. You can set max_tokens high enough to accommodate long reasoning without worrying about a token meter spinning. If you need structured output, you can combine reasoning with JSON mode or function calling to extract the final answer from the generated trace.

When to Use Explicit Reasoning Architectures

Not every task benefits from chain-of-thought overhead. Simple classification or retrieval is often faster and cheaper with a compact model like Llama 3.3 70B or Qwen 3 32B in direct-answer mode. Reserve DeepSeek R1, Kimi K2 Thinking, and GLM 5 for tasks where accuracy matters more than latency: formal verification, complex mathematics, multi-step coding, and agent planning. Oxlo.ai’s catalog covers both ends of this spectrum, so you can route queries to a reasoning architecture only when the problem demands it.

Conclusion

Chain-of-thought reasoning is no longer a prompt engineering trick. It is an architectural axis that influences model training, inference cost, and system design. Platforms that treat reasoning as a first-class workload, with predictable pricing and broad model coverage, give developers the freedom to experiment without budget surprises. Oxlo.ai’s request-based pricing and full OpenAI SDK compatibility remove the friction from deploying deep reasoning models at scale. If you are building agents, coding assistants, or research tools that rely on extended reasoning, you can explore the model catalog and pricing at https://oxlo.ai/pricing.

Building Language Models from Scratch with LLM and Transfer Learning

shashank ms — Wed, 09 Sep 2026 01:34:33 +0000

We will build a domain-specific support ticket classifier using few-shot transfer learning with an Oxlo.ai-hosted LLM. This approach is for teams that need customized model behavior immediately without managing GPU fine-tuning infrastructure.

What you'll need

Python 3.10 or newer
An Oxlo.ai API key from https://portal.oxlo.ai
The OpenAI SDK: pip install openai

Step 1: Verify connectivity

Set up the OpenAI-compatible client and confirm you can reach Oxlo.ai. I use llama-3.3-70b here because it follows formatting instructions reliably.

from openai import OpenAI
import os

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key=os.environ.get("OXLO_API_KEY")
)

response = client.chat.completions.create(
    model="llama-3.3-70b",
    messages=[{"role": "user", "content": "Say OK"}],
)
print(response.choices[0].message.content)

Step 2: Design the transfer learning prompt

Instead of training new weights from scratch, we adapt a frozen foundation model by embedding task-specific examples in the system prompt. This is in-context transfer learning. The prompt below defines the classification schema and the few-shot adaptation layer.

SYSTEM_PROMPT = """You are a support ticket classifier. Read the customer message and classify it into exactly one category.

Categories:
- billing: payment, invoices, refunds
- technical: bugs, errors, feature questions
- account: login, security, profile changes

Respond ONLY with a JSON object in this format:
{"category": "<category>", "urgency": "low|medium|high", "reason": "<one sentence>"}

Examples:
User: I was charged twice for my subscription this month.
Assistant: {"category": "billing", "urgency": "high", "reason": "Duplicate charge requires immediate refund review."}

User: How do I reset my password?
Assistant: {"category": "account", "urgency": "medium", "reason": "Standard account recovery request."}

User: The API returns a 500 error when I post to /v1/items.
Assistant: {"category": "technical", "urgency": "high", "reason": "Server error indicates a bug in production."}
"""

Step 3: Build the inference wrapper

We enforce structured output by setting response_format to json_object. The function below assembles the system prompt and the new user message into a single Oxlo.ai API call.

import json

def classify_ticket(user_message: str) -> dict:
    response = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": user_message},
        ],
        response_format={"type": "json_object"},
        temperature=0.1,
    )
    raw = response.choices[0].message.content
    return json.loads(raw)

Step 4: Test on unseen inputs

These tickets are not in the few-shot examples, so they test whether the model has actually transferred the classification concept or is simply memorizing.

test_tickets = [
    "My invoice shows the wrong tax rate. Can you fix it?",
    "I get a timeout every time I upload a file larger than 10MB.",
    "I want to update my email address but the confirmation link is broken.",
    "Do you offer yearly billing instead of monthly?",
]

for ticket in test_tickets:
    result = classify_ticket(ticket)
    print(f"Ticket: {ticket}")
    print(f"Result: {json.dumps(result, indent=2)}")
    print()

Run it

Here is the complete script. Save it as ticket_classifier.py, set your OXLO_API_KEY, and run it.

from openai import OpenAI
import os
import json

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key=os.environ.get("OXLO_API_KEY")
)

SYSTEM_PROMPT = """You are a support ticket classifier. Read the customer message and classify it into exactly one category.

Categories:
- billing: payment, invoices, refunds
- technical: bugs, errors, feature questions
- account: login, security, profile changes

Respond ONLY with a JSON object in this format:
{"category": "<category>", "urgency": "low|medium|high", "reason": "<one sentence>"}

Examples:
User: I was charged twice for my subscription this month.
Assistant: {"category": "billing", "urgency": "high", "reason": "Duplicate charge requires immediate refund review."}

User: How do I reset my password?
Assistant: {"category": "account", "urgency": "medium", "reason": "Standard account recovery request."}

User: The API returns a 500 error when I post to /v1/items.
Assistant: {"category": "technical", "urgency": "high", "reason": "Server error indicates a bug in production."}
"""

def classify_ticket(user_message: str) -> dict:
    response = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": user_message},
        ],
        response_format={"type": "json_object"},
        temperature=0.1,
    )
    raw = response.choices[0].message.content
    return json.loads(raw)

if __name__ == "__main__":
    test_tickets = [
        "My invoice shows the wrong tax rate. Can you fix it?",
        "I get a timeout every time I upload a file larger than 10MB.",
        "I want to update my email address but the confirmation link is broken.",
        "Do you offer yearly billing instead of monthly?",
    ]

    for ticket in test_tickets:
        result = classify_ticket(ticket)
        print(f"Ticket: {ticket}")
        print(f"Result: {json.dumps(result, indent=2)}")
        print()

Example output:

Ticket: My invoice shows the wrong tax rate. Can you fix it?
Result: {
  "category": "billing",
  "urgency": "high",
  "reason": "Incorrect tax calculation affects invoice accuracy."
}

Ticket: I get a timeout every time I upload a file larger than 10MB.
Result: {
  "category": "technical",
  "urgency": "medium",
  "reason": "Upload timeout suggests a performance issue."
}

Ticket: I want to update my email address but the confirmation link is broken.
Result: {
  "category": "account",
  "urgency": "medium",
  "reason": "Broken confirmation link blocks profile update."
}

Ticket: Do you offer yearly billing instead of monthly?
Result: {
  "category": "billing",
  "urgency": "low",
  "reason": "General billing plan inquiry with no immediate impact."
}

Wrap-up

If your ticket volume grows, expand the few-shot examples in the system prompt or switch to qwen-3-32b for multilingual classification. Because Oxlo.ai charges a flat rate per request rather than per token, adding more context examples does not increase your inference cost, which makes this transfer learning pattern especially practical at scale. See the details at https://oxlo.ai/pricing.

LLM Inference Platforms with Request-Based Pricing

shashank ms — Wed, 09 Sep 2026 01:32:46 +0000

Most AI inference platforms bill by the token. Input tokens, output tokens, and sometimes context-window premiums all feed into a variable cost that is hard to predict and harder to optimize. Request-based pricing flips the model. You pay one flat fee per API call, regardless of whether you send a ten-word prompt or a ten-thousand-word document. For teams running long-context retrieval, agentic loops, or large-batch processing, this predictability is not just a billing convenience. It is an architectural advantage.

What Is Request-Based Pricing?

Under a request-based model, the unit of cost is the HTTP request, not the token. Whether your payload is 512 tokens or 128,000 tokens, the price of the call stays the same. This decouples your infrastructure budget from your prompt engineering decisions.

Oxlo.ai is a developer-first AI inference platform built on this exact model. It charges one flat cost per API request regardless of prompt length. Unlike token-based providers such as Together AI, Fireworks AI, OpenRouter, Replicate, and Anyscale, cost does not scale with input length. That distinction matters the moment you start passing large contexts, conversation histories, or multi-modal inputs to the model.

Where Token Costs Spiral

Token-based pricing is straightforward for short queries, but it creates friction in several real-world patterns:

Retrieval-Augmented Generation (RAG): Injecting large document chunks into the context window increases input tokens linearly.
Agentic workflows: Each tool call and observation appends more text to the conversation history. Over multiple turns, the prompt bloats.
Code generation: Supplying a model with multiple files, dependency trees, or error logs can consume tens of thousands of tokens before a single completion token is generated.

In these scenarios, the majority of your bill can come from simply stating the problem, not from solving it.

How Request-Based Pricing Changes the Math

When cost is tied to the request boundary, a 1,000-token prompt and a 100,000-token prompt are priced identically. This makes Oxlo.ai significantly cheaper for long-context and agentic workloads. You can pass full files, long conversation histories, or extensive system prompts without watching a meter spin.

The financial difference can be dramatic. For workloads that regularly fill large context windows, request-based pricing can be 10-100x cheaper than token-based alternatives. More importantly, it turns variable OpEx into a fixed unit cost, which makes capacity planning and margin control easier for product teams.

Platform Comparison

The inference market is split between token-based aggregators and request-first platforms. Token-based providers meter every input and output token, which favors short, chat-style interactions but penalizes research, coding agents, and document analysis.

Oxlo.ai offers 45+ open-source and proprietary models across 7 categories, all behind a single API that is fully OpenAI SDK compatible. The catalog includes:

LLMs / chat and reasoning: Qwen 3, Llama 3/4, DeepSeek R1/V3, Kimi K2.x, GPT-Oss, Mistral, GLM 5, Minimax
Code: Qwen 3 Coder 30B, DeepSeek Coder, Oxlo.ai Coder Fast
Vision: Gemma 3 27B, Kimi VL A3B
Image generation: Oxlo.ai Image Pro and Ultra, Flux.1, SDXL, Stable Diffusion 3.5
Audio: Whisper Large v3 / Turbo / Medium, Kokoro 82M text-to-speech
Embeddings: BGE-Large, E5-Large
Object detection: YOLOv9, YOLOv11

Supported features include streaming responses, function calling and tool use, JSON mode, vision input, and multi-turn conversations. Endpoints cover chat/completions, embeddings, images/generations, audio/transcriptions, and audio/speech. There are no cold starts on popular models, so latency is consistent from the first request.

Getting Started with Oxlo.ai

Because Oxlo.ai is a fully OpenAI API compatible drop-in replacement, you can switch providers without rewriting client code. Change the base URL and API key, and existing Python, Node.js, or cURL scripts continue to work.

import openai

client = openai.OpenAI(
    api_key="YOUR_OXLO_API_KEY",
    base_url="https://api.oxlo.ai/v1"
)

response = client.chat.completions.create(
    model="llama-3.3-70b",
    messages=[
        {"role": "system", "content": "You are a senior software engineer."},
        {"role": "user", "content": "Refactor this 500-line module to use async/await."}
    ],
    stream=True
)

for chunk in response:
    print(chunk.choices[0].delta.content or "", end="")

In this example, the prompt could be a few tokens or a full file dump. Under Oxlo.ai's request-based pricing, the cost of the call remains flat. You can explore the exact plans on the Oxlo.ai pricing page. The Free tier offers $0 per month, 60 requests per day, and access to 16+ free models, plus a 7-day full-access trial. Paid tiers include Pro at $80 per month with 1,000 requests per day, Premium at $350 per month with 5,000 requests per day and priority queue, and Enterprise with custom unlimited usage and dedicated GPUs.

When to Choose Request-Based Pricing

Request-based pricing is not a universal cure. If your workload consists of very short, uniform prompts, the difference between per-token and per-request billing may be negligible. But if your architecture involves any of the following, the model is worth evaluating:

Long-context RAG with large retrieved chunks
Multi-step agents that accumulate conversation state
Batch processing of documents, codebases, or media
Unpredictable prompt lengths that make token budgeting impossible

For these patterns, Oxlo.ai provides a predictable, developer-first alternative to token-based metering. With broad model coverage, OpenAI SDK compatibility, and no cold starts, it is built for teams that want to ship agentic and long-context features without renegotiating their cost model on every release.

High-Throughput and Low-Latency LLM Inference Optimization Techniques

shashank ms — Tue, 08 Sep 2026 23:36:13 +0000

Production LLM inference is a balancing act. Throughput, measured in tokens per second across all users, must climb while latency, the time to first token and inter-token latency for an individual user, must fall. Achieving both requires moving beyond naive single-request serving to a stack of software and hardware optimizations. The techniques below are what separate prototype APIs from infrastructure that can serve millions of daily requests at scale.

Batching Strategies

The simplest way to improve GPU utilization is to process multiple requests together. Static batching groups a fixed set of prompts, but it is inefficient because sequences complete at different lengths. The entire batch must wait for the longest generation, leaving compute units idle.

Dynamic batching improves on this by grouping requests that arrive within a short window. While better, it still suffers from the same tail-latency problem.

Continuous batching, also called in-flight batching, solves this at the iteration level. Instead of waiting for every sequence in a batch to finish, the scheduler swaps out completed sequences and swaps in new requests after every forward pass. This keeps the GPU saturated and is now standard in production engines like vLLM and TGI.

KV Cache Management and PagedAttention

For autoregressive transformers, the key-value cache is the dominant memory consumer. A naive implementation preallocates a contiguous buffer sized to the model's maximum context length for every request. This wastes memory on short prompts and creates internal fragmentation, limiting batch size.

PagedAttention, introduced by vLLM, treats the KV cache like an operating system's virtual memory. The cache is divided into fixed-size blocks that are allocated non-contiguously and mapped via a block table. When a sequence generates a new token, it only needs a new block, not a full reallocation. This allows significantly higher batch sizes and throughput on the same hardware.

Quantization and Compression

Quantization reduces the precision of weights and activations, shrinking model size and accelerating matrix multiplications. INT8 quantization typically recovers near-full accuracy with a 2x memory reduction. More aggressive schemes like GPTQ, AWQ, and GGUF push weights to INT4, enabling large models to fit on fewer GPUs.

The tradeoff is accuracy versus latency. INT4 can introduce perplexity degradation for reasoning-heavy models. For production APIs, INT8 or FP8 mixed precision often offers the best balance on modern NVIDIA hardware with dedicated tensor cores for low-precision math.

Speculative Decoding

Speculative decoding reduces per-request latency by using a small draft model to generate candidate tokens, which a larger target model then verifies in parallel. If the draft is accurate, multiple tokens are accepted per forward pass of the large model. This is especially effective for code generation and structured outputs where local patterns are predictable.

The overhead is memory, you must host both models, and the draft model must share the target model's tokenizer. When throughput headroom exists, speculative decoding can cut time-to-final-token without sacrificing accuracy.

Continuous Batching and Scheduling

Modern inference engines use iteration-level scheduling to maximize throughput. Beyond simple continuous batching, advanced schedulers implement prefix caching. When multiple users share a system prompt or when an agent performs multi-turn reasoning, the precomputed key-value states for the shared prefix are stored and reused. This avoids redundant computation and dramatically reduces time-to-first-token for long conversations.

Function calling and JSON mode, features common in agentic workloads, also benefit from optimized scheduling. Constrained decoding can be fused into the sampling loop so that token generation adheres to a grammar or schema without expensive post-hoc filtering.

Model Parallelism and Tensor Sharding

When a model exceeds the memory of a single GPU, parallelism strategies become mandatory. Tensor parallelism splits individual layers across devices, requiring high-bandwidth interconnects like NVLink to keep latency low. Pipeline parallelism assigns contiguous layers to different GPUs, but introduces bubble overhead unless microbatching is tuned carefully.

For Mixture-of-Experts architectures like DeepSeek R1 671B MoE or GLM 5, expert parallelism routes tokens to specific GPU workers. Efficient all-to-all communication patterns determine whether an MoE model achieves its theoretical throughput or becomes network-bound.

Platform Considerations for Production Workloads

Implementing these optimizations in-house requires maintaining custom CUDA kernels, scheduling logic, and multi-node orchestration. Most engineering teams will see better ROI by routing inference to a managed platform that already implements continuous batching, PagedAttention, and quantization.

Oxlo.ai is a developer-first AI inference platform built for these production constraints. It offers fully OpenAI SDK compatible APIs, so switching requires only a base URL change.

import openai

client = openai.OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key="YOUR_OXLO_API_KEY"
)

response = client.chat.completions.create(
    model="deepseek-r1-671b",
    messages=[{"role": "user", "content": "Explain speculative decoding"}],
    stream=True
)

for chunk in response:
    print(chunk.choices[0].delta.content, end="")

Unlike token-based providers such as Together AI, Fireworks AI, OpenRouter, Replicate, or Anyscale, Oxlo.ai uses request-based pricing. You pay one flat cost per API request regardless of prompt length. For long-context and agentic workloads, this model eliminates the cost explosion tied to input tokens and can be significantly cheaper. Oxlo.ai also offers no cold starts on popular models, streaming responses, function calling, JSON mode, and vision support across 45+ open-source and proprietary models.

Pricing is transparent. The Free plan includes 60 requests per day and access to 16+ free models, while paid tiers scale to dedicated GPU clusters for Enterprise workloads. For exact rates, see the Oxlo.ai pricing page.

Conclusion

High-throughput, low-latency inference is not the result of any single trick. It is a stack of batching, memory management, quantization, parallelism, and scheduling optimizations. For teams running production chat, coding agents, or vision pipelines, building this stack from scratch diverts engineering resources from product development.

Oxlo.ai provides an optimized inference backend with predictable request-based pricing, OpenAI SDK compatibility, and a broad model catalog including Llama 3.3 70B, Qwen 3 32B, DeepSeek V4 Flash, and Kimi K2.6. If your workloads are growing in context length or complexity, it is worth evaluating a platform that aligns cost with requests rather than tokens.

LLM Sentiment Analysis and Text Classification Guide

shashank ms — Tue, 08 Sep 2026 23:34:39 +0000

Support teams drown in unstructured feedback. In this tutorial, we will build a working sentiment and topic classifier that labels customer messages in a single API call, then batch-processes a backlog using Oxlo.ai. The entire pipeline uses the OpenAI SDK with an Oxlo.ai base URL, so there is no new client library to learn.

What you'll need

Python 3.10 or newer. An Oxlo.ai API key from https://portal.oxlo.ai. The OpenAI SDK installed with pip install openai.

Step 1: Configure the Oxlo.ai client

I keep credentials in an environment variable so I do not accidentally commit keys. The Oxlo.ai client is a drop-in replacement for the standard OpenAI client.

import os
from openai import OpenAI

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key=os.getenv("OXLO_API_KEY", "YOUR_OXLO_API_KEY"),
)

Step 2: Define the system prompt

The trick to reliable classification is telling the model exactly what valid JSON looks like. I use a strict system prompt so every response follows the same structure.

SYSTEM_PROMPT = """You are a classification engine. Analyze the user message and return ONLY a JSON object with this exact structure:

{
  "sentiment": "negative" | "neutral" | "positive",
  "topic": "billing" | "bug" | "feature_request" | "other",
  "confidence": 0.0 to 1.0,
  "reasoning": "one sentence explaining the label"
}

Rules:
- sentiment must be one of the three allowed strings.
- topic must be one of the four allowed strings.
- confidence is your certainty score.
- Do not include markdown, explanations, or text outside the JSON."""

Step 3: Build the classifier function

I wrap the API call in a small function so I can swap models later. I enable JSON mode to enforce valid output, and I default to llama-3.3-70b because it handles mixed instructions cleanly. If your data is multilingual, swap the model string to qwen-3-32b.

import json

def classify_message(text: str, model: str = "llama-3.3-70b"):
    response = client.chat.completions.create(
        model=model,
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": text},
        ],
        response_format={"type": "json_object"},
        temperature=0.1,
    )

    raw = response.choices[0].message.content
    return json.loads(raw)

Step 4: Process a batch of tickets

Most real workloads are not single messages. I simulate a small queue and run each item through the classifier, collecting results in a list. Because Oxlo.ai uses flat request-based pricing, long tickets do not inflate cost the way token-based metering does. See details at https://oxlo.ai/pricing.

tickets = [
    "I was charged twice last month and your dashboard is broken. Fix this now.",
    "Love the new export feature. Saves me an hour every week.",
    "How do I change my notification settings? I looked everywhere.",
    "The API returns a 500 error when I send payloads over 1 MB. Here is the curl...",
    "Please add dark mode. It is hard to use at night.",
]

results = []
for t in tickets:
    try:
        label = classify_message(t)
        results.append({"text": t, "label": label})
    except Exception as e:
        results.append({"text": t, "error": str(e)})

print(json.dumps(results, indent=2))

Step 5: Filter by confidence

Raw labels are not enough. I filter out anything below a confidence threshold so a human can review the edge cases.

CONFIDENCE_THRESHOLD = 0.85

flagged = []
clean = []

for r in results:
    if "error" in r:
        flagged.append(r)
        continue

    conf = r["label"].get("confidence", 0)
    if conf < CONFIDENCE_THRESHOLD:
        flagged.append(r)
    else:
        clean.append(r)

print(f"Auto-approved: {len(clean)}")
print(f"Needs review: {len(flagged)}")

for item in clean:
    print(f"[{item['label']['sentiment']}] {item['label']['topic']} -> {item['text'][:50]}...")

Run it

Save everything in classify.py, export your key, and run python classify.py. Here is what my last run looked like.

$ export OXLO_API_KEY="sk-oxlo.ai-..."
$ python classify.py

Auto-approved: 4
Needs review: 1

[negative] billing -> I was charged twice last month and your dashboard...
[positive] feature_request -> Love the new export feature. Saves me an ho...
[neutral] other -> How do I change my notification settings? I loo...
[negative] bug -> The API returns a 500 error when I send payloa...

Next steps

Two ways to extend this immediately. First, wrap the classify_message function in a FastAPI endpoint so your support stack can call it live. Second, if you are classifying long conversation threads instead of single messages, switch to kimi-k2.6 or deepseek-v3.2. Both handle extended context and reasoning well on Oxlo.ai.

Optimizing LLM Inference for Edge Devices

shashank ms — Tue, 08 Sep 2026 23:32:39 +0000

Edge deployment of large language models forces a direct confrontation with physics. Memory bandwidth, thermal limits, and battery life turn every generation cycle into a trade-off between accuracy and feasibility. Most teams start by shrinking models, pruning weights, or deploying dedicated NPUs. Yet the edge is not a monolith. For many production workloads, the most reliable optimization is not running the model locally at all, but routing inference to a cloud backend engineered for low latency and predictable cost. Oxlo.ai provides a request-based inference platform that removes token-length pricing uncertainty, making it a natural fit for edge agents that ship variable context back to the cloud.

The Edge Inference Bottleneck

On-device inference is almost always memory-bound, not compute-bound. A 7B parameter model in FP16 requires roughly 14 GB of RAM just for weights, before accounting for the KV cache, activation buffers, and OS overhead. At the edge, DRAM is scarce, LPDDR bandwidth is a fraction of server GDDR, and every watt draws from a battery. Small batch sizes and single-user sessions mean GPUs and NPUs sit underutilized while memory channels saturate. The result is high time-to-first-token and per-token latency that degrades sharply as context length grows.

Quantization and Model Compression

Quantization is the first tool most engineers reach for. Moving from FP16 to INT8 halves storage and doubles effective bandwidth. INT4 and formats like GPTQ and AWQ push further by compressing weights and using grouped quantization to recover accuracy. For vision and audio encoders, pruning and knowledge distillation can shrink student models to a fraction of the teacher size. These techniques work well for classification, extraction, and small generative tasks on modern phone NPUs. However, once the model exceeds the available RAM on the target device, even aggressive compression cannot salvage on-device execution. At that threshold, offloading becomes the only viable path to run frontier-class models.

KV Cache Management and Memory Boundaries

During autoregressive decoding, the KV cache grows linearly with sequence length and layer count. For a 32-layer model and a 32K context, the cache can balloon to multiple gigabytes, easily overwhelming edge memory budgets. On-device frameworks mitigate this through quantized caches, sliding-window attention, and prompt caching, but these are bounded by the physical memory pool. When the cache exceeds capacity, the system either crashes or falls back to CPU paging, which collapses latency. Cloud inference platforms can host the full cache on high-bandwidth server memory, but traditional token-based pricing penalizes long contexts. Oxlo.ai avoids this trade-off entirely with flat per-request pricing, so edge clients can stream large sensor logs or multi-turn histories without watching metered tokens accumulate.

Batching and Request Scheduling

Edge silicon is optimized for throughput under large batches, yet edge workloads are typically asynchronous and single-tenant. A local model serving one user at batch size 1 leaves massive compute potential idle. Continuous batching and in-flight request scheduling are solutions that only exist in data-center inference engines such as vLLM and TensorRT-LLM. By routing requests to a cloud API, edge devices effectively borrow a multi-tenant scheduler without burning local power. Oxlo.ai runs popular models with no cold starts, so an edge device can open a connection, send a payload, and receive a streamed response without the warmup latency that plagues serverless token-based platforms.

Cloud Offload as an Edge Strategy

Treating the cloud as an extension of the edge is not a compromise. It is an architecture decision. For agents that roam across Wi-Fi, 5G, and LoRa, the key requirements are consistent API behavior, low latency, and cost predictability. Oxlo.ai meets these with an OpenAI-compatible endpoint at https://api.oxlo.ai/v1, which means existing edge clients using the Python or Node.js SDKs can switch base URLs without rewriting logic. Because Oxlo.ai charges one flat cost per request regardless of prompt length, an edge agent that sends a 10K token system prompt plus image context pays the same as a one-sentence query. For long-context and agentic workloads, request-based pricing can be 10-100x cheaper than token-based alternatives, a gap that widens as edge agents accumulate memory. Current plans are listed at https://oxlo.ai/pricing.

The platform offers 45+ models across seven categories, including lightweight code models like Qwen 3 Coder 30B, vision models such as Kimi VL A3B, and multilingual options like Qwen 3 32B, so edge applications can select a capability tier without managing separate deployments.

Edge Gateway Example

A practical pattern is the edge gateway: a thin local service that pre-processes sensor data, decides whether to run a tiny local model or escalate to the cloud, and streams the result back to the device. Below is a minimal Python gateway that routes complex reasoning to Oxlo.ai while keeping simple intent classification local.

import os
from openai import OpenAI

# Configure the client for Oxlo.ai
client = OpenAI(
    api_key=os.getenv("OXLO_API_KEY"),
    base_url="https://api.oxlo.ai/v1"
)

def route_request(context: str, local_confidence: float) -> str:
    # Fallback threshold for local model
    if local_confidence > 0.9 and len(context) < 200:
        return run_local_tiny_llm(context)
    
    # Offload heavy or long-context work to Oxlo.ai
    response = client.chat.completions.create(
        model="qwen3-32b",
        messages=[
            {"role": "system", "content": "You are an edge reasoning agent."},
            {"role": "user", "content": context}
        ],
        stream=True,
        max_tokens=512
    )
    
    # Stream tokens back to the edge client
    return "".join(chunk.choices[0].delta.content or "" for chunk in response)

# Example: sensor fusion log with high token count
sensor_log = "[CAMERA] object_detected:3 ..."
result = route_request(sensor_log, local_confidence=0.4)
print(result)

This pattern keeps the edge device responsive. Simple tasks never leave the hardware, while complex reasoning benefits from server-class memory and Oxlo.ai's flat request pricing.

Conclusion

Optimizing LLM inference for the edge is not solely a matter of squeezing

Understanding Large Language Model Training Datasets

shashank ms — Tue, 08 Sep 2026 21:36:18 +0000

We're going to build a command-line Training Dataset Profiler that ingests a raw JSONL fine-tuning dump, validates structure, flags quality issues, and produces a human-readable summary. If you have ever downloaded a "cleaned" dataset from Hugging Face only to find empty responses and leaked emails inside, this tool saves you from training on garbage.

What you'll need

Python 3.10 or newer
An Oxlo.ai API key from https://portal.oxlo.ai
The OpenAI SDK: pip install openai

Step 1: Bootstrap the project and Oxlo.ai client

First we initialize the OpenAI-compatible client pointing at Oxlo.ai. I keep my key in an environment variable so it does not end up in shell history.

import os
import json
from openai import OpenAI

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key=os.getenv("OXLO_API_KEY", "YOUR_OXLO_API_KEY")
)

Step 2: Create a dirty sample dataset

To make this reproducible without downloading multi-gigabyte files, we will synthesize a small JSONL file that mimics real instruction-tuning data. I have sprinkled in duplicates, PII, and malformed entries so the profiler has something to catch.

def create_sample_dataset(path: str = "sample_data.jsonl"):
    records = [
        {"instruction": "What is the capital of France?", "input": "", "output": "Paris"},
        {"instruction": "Write a Python hello world", "input": "", "output": "print('hello world')"},
        {"instruction": "What is the capital of France?", "input": "", "output": "Paris"},
        {"instruction": "Email me the report", "input": "", "output": "Sure, I will send it to alice@example.com tomorrow."},
        {"instruction": "", "input": "", "output": ""},
        {"instruction": "Explain quantum computing", "input": "", "output": "Quantum computing uses qubits. " * 50},
        {"instruction": "Fix this bug", "input": "def foo():\n    pass", "output": "You should use a better function name."},
    ]
    with open(path, "w", encoding="utf-8") as f:
        for r in records:
            f.write(json.dumps(r, ensure_ascii=False) + "\n")
    return path

create_sample_dataset()

Step 3: Define the profiler agent

The profiler is just a system prompt. We treat the LLM as a structured analyst that receives raw JSON and returns findings as JSON. You can edit this prompt to add domain-specific rules, such as rejecting outputs under a certain length or flagging specific keywords.

SYSTEM_PROMPT = """You are a Training Dataset Profiler. Your job is to analyze raw JSONL records from an LLM fine-tuning dataset and return a concise JSON object with exactly these keys:

- schema_detected: string, one of ["alpaca", "sharegpt", "custom", "unknown"]
- total_records_analyzed: integer
- issues: array of objects, each with keys "severity" ("high", "medium", "low"), "category" ("duplicate", "pii", "empty_field", "formatting", "quality"), "record_index": integer, "description": string
- summary: string, a two-sentence human-readable summary

Be strict. If an output contains an email address, phone number, or API key, flag it as pii with high severity. If an instruction-output pair is identical to a previous pair, flag it as duplicate. If any required field is empty, flag it as empty_field."""

Step 4: Inspect schema and format

Before we run statistics, we need to know if this is Alpaca, ShareGPT, or a custom format. We send the first three records to Oxlo.ai with a focused user message. I am using Llama 3.3 70B here because it handles general-purpose structured analysis reliably.

def detect_schema(file_path: str) -> str:
    with open(file_path, "r", encoding="utf-8") as f:
        lines = [json.loads(f.readline()) for _ in range(3)]
    
    user_message = (
        "Analyze these sample records and tell me the dataset schema. "
        "Return only one word: alpaca, sharegpt, or custom.\n\n"
        + json.dumps(lines, indent=2)
    )
    
    response = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": user_message},
        ],
    )
    return response.choices[0].message.content.strip()

schema = detect_schema("sample_data.jsonl")
print(f"Detected schema: {schema}")

Step 5: Surface data-quality issues

Now we feed a larger slice of the dataset and ask the agent to flag exact problems. Because Oxlo.ai uses flat per-request pricing, sending a batched context of several records costs the same whether the snippet is 1K or 10K tokens. That makes exploratory profiling predictable when you are scanning long-context training dumps.

def profile_records(file_path: str, limit: int = 20) -> dict:
    with open(file_path, "r", encoding="utf-8") as f:
        records = [json.loads(line) for line in f][:limit]
    
    user_message = (
        f"Analyze these {len(records)} records and return your findings as JSON. "
        "Do not wrap the JSON in markdown code fences.\n\n"
        + json.dumps(records, indent=2, ensure_ascii=False)
    )
    
    response = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": user_message},
        ],
    )
    raw = response.choices[0].message.content.strip()
    if raw.startswith("

```"):
        raw = raw.split("```

")[1].replace("json", "").strip()
    return json.loads(raw)

report = profile_records("sample_data.jsonl")
print(json.dumps(report, indent=2))

Step 6: Generate an executive summary

Finally, we ask the model to synthesize the structured report into a short markdown summary we can paste into a data card or pull request description.

def summarize(report: dict) -> str:
    user_message = (
        "Turn this structured dataset report into a concise markdown summary "
        "suitable for a data card. Include bullet points for high-severity issues.\n\n"
        + json.dumps(report, indent=2)
    )
    
    response = client.chat.completions.create(
        model="qwen-3-32b",
        messages=[
            {"role": "system", "content": "You write concise data-quality summaries in markdown."},
            {"role": "user", "content": user_message},
        ],
    )
    return response.choices[0].message.content.strip()

markdown_summary = summarize(report)
print(markdown_summary)

Run it

Putting it all together, the full script loads the dirty sample, detects the schema, profiles the records, and prints the markdown summary. Here is the main entry point and an example of what you should see in your terminal.

if __name__ == "__main__":
    create_sample_dataset()
    schema = detect_schema("sample_data.jsonl")
    print(f"\nSchema: {schema}\n")
    report = profile_records("sample_data.jsonl")
    print("\nStructured report:")
    print(json.dumps(report, indent=2))
    print("\nExecutive summary:")
    print(summarize(report))

Example output:

Schema: alpaca

Structured report:
{
  "schema_detected": "alpaca",
  "total_records_analyzed": 7,
  "issues": [
    {"severity": "high", "category": "duplicate", "record_index": 2, "description": "Identical instruction-output pair to record 0"},
    {"severity": "high", "category": "pii", "record_index": 3, "description": "Contains email address alice@example.com"},
    {"severity": "high", "category": "empty_field", "record_index": 4, "description": "All fields are empty"},
    {"severity": "medium", "category": "quality", "record_index": 5, "description": "Output is excessively repetitive"}
  ],
  "summary": "Dataset appears to follow Alpaca schema. Found 1 duplicate, 1 PII leak, 1 empty record, and 1 low-quality repetitive output."
}

Executive summary:
- **Schema:** Alpaca-style instruction tuning
- **High Severity Issues:**
  - Record 2 is a duplicate of Record 0
  - Record 3 leaks an email address
  - Record 4 is completely empty
- **Recommendation:** Remove duplicates and PII before training.

Wrap-up and next steps

This profiler is a starting point, not a production pipeline. A concrete next step is to wire it into a CI job that rejects commits when PII or duplicates are detected above a threshold. You could also parallelize it across shards of a large corpus.

If you scale this to massive datasets, Oxlo.ai's request-based pricing becomes a clear win. Profiling a batch of 100K tokens costs the same flat rate as a 1K token ping, so you can stuff large context windows for deep analysis without the bill scaling linearly. For details, see https://oxlo.ai/pricing.