DEV Community: shashank ms

Anomaly Detection with LLM Inference: Best Practices and Applications

shashank ms — Mon, 27 Jul 2026 07:36:27 +0000

Anomaly detection has evolved beyond statistical thresholds and isolation forests. Modern systems generate heterogeneous, high-dimensional telemetry, logs, and event streams that defeat traditional heuristics. Large language models can interpret unstructured context, correlate semantic signals across disparate sources, and generate human-readable root-cause narratives. The limiting factor is rarely the model architecture itself, but the inference infrastructure: context limits, latency, and pricing models that penalize you for feeding the model enough information to make an accurate determination.

Why LLMs for Anomaly Detection

Classical anomaly detection relies on fixed schemas and statistical distance metrics. Production failures rarely respect those boundaries. A memory leak in a microservice can manifest as a subtle shift in log vocabulary, a delayed Kafka offset, and an anomalous latency tail simultaneously. LLMs excel at this type of cross-domain correlation because they reason over unstructured text natively. They can compare a current incident narrative against historical postmortems, identify semantic drift in application logs, and produce actionable summaries for on-call engineers. The challenge is operationalizing this capability at scale without letting inference costs dominate your observability budget.

Inference Architecture Patterns

There are three common patterns for integrating LLMs into anomaly detection pipelines.

Direct classification. Send raw logs, metrics as text, or event descriptions directly to a chat model with a structured output schema. This works best when the anomaly is semantic, such as detecting novel error types or policy violations.

Embedding plus LLM. Use embedding models to compress historical baselines into vector space. Flag vectors that exceed a distance threshold, then use an LLM to investigate only the flagged outliers. Oxlo.ai offers BGE-Large and E5-Large through the embeddings endpoint, so you can run the entire pipeline on a single platform.

Agentic investigation. Deploy an agent that uses function calling to query additional data sources, iterate on hypotheses, and confirm anomalies before alerting. This reduces false positives but requires multiple tool calls and long context retention. Oxlo.ai supports function calling, tool use, and multi-turn conversations, and the request-based pricing model keeps agentic loops predictable even when context grows.

Long Context and Cost Efficiency

Anomaly detection is a long-context workload. A single incident may require thousands of log lines, distributed trace spans, and configuration snippets to diagnose accurately. Under token-based pricing, each additional line increases cost. Oxlo.ai flips this model: one flat cost per API request regardless of prompt length. Unlike token-based providers such as Together AI, Fireworks AI, OpenRouter, Replicate, and Anyscale, cost does not scale with input length, so Oxlo.ai is significantly cheaper for long-context and agentic workloads.

This pricing structure changes architectural decisions. You can afford to dump complete trace buffers rather than aggressively summarizing or filtering logs upstream. Models like DeepSeek V4 Flash provide a 1 million token context window, while Kimi K2.6 offers a 131K context window with advanced reasoning and agentic coding capabilities. Both are available on Oxlo.ai with no cold starts, so you can send large payloads on demand without latency penalties.

Code Example: Structured Detection with JSON Mode

The following example uses the OpenAI SDK with Oxlo.ai to analyze a system trace. JSON mode forces a machine-readable response that your alerting stack can consume directly.

import os
import openai

client = openai.OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key=os.environ.get("OXLO_API_KEY")
)

trace_buffer = """
[2024-05-21T14:32:01Z] svc-auth latency=12ms status=200
[2024-05-21T14:32:02Z] svc-auth latency=340ms status=503
[2024-05-21T14:32:02Z] db-primary connection_pool_exhausted
[2024-05-21T14:32:03Z] cache-redis timeout
...
"""  # Production buffers may contain thousands of lines

response = client.chat.completions.create(
    model="deepseek-v3.2",
    messages=[
        {
            "role": "system",
            "content": (
                "You are an anomaly detection engine. Analyze the trace. "
                "Respond with strict JSON only: anomaly_detected (boolean), "
                "severity (low, medium, high), root_cause (string), remediation (string)."
            )
        },
        {
            "role": "user",
            "content": f"System trace:\n{trace_buffer}"
        }
    ],
    response_format={"type": "json_object"}
)

print(response.choices[0].message.content)

Because Oxlo.ai is fully OpenAI SDK compatible, this is a drop-in replacement. Change the base URL and API key, and your existing Python, Node.js, or cURL workflows remain intact.

Model Selection for Detection Tasks

Different anomaly detection tasks demand different model capabilities. Oxlo.ai hosts over 45 models across 7 categories, allowing you to match the tool to the signal.

Deep reasoning and root cause analysis: DeepSeek R1 671B MoE and Kimi K2.6 handle complex, multi-step causal chains. Use these when the anomaly spans multiple services and requires reasoning about architecture dependencies.
Massive context ingestion: DeepSeek V4 Flash combines a 1 million token context window with efficient MoE inference. Ideal for analyzing complete unabridged log exports or lengthy trace waterfalls.
Multilingual or mixed-format logs: Qwen 3 32B provides strong multilingual reasoning and agent workflow support for global infrastructure.
General-purpose fast detection: Llama 3.3 70B offers low-latency verdicts for high-volume streaming pipelines.
Prototyping and cost-sensitive detection: DeepSeek V3.2 supports coding and reasoning tasks and is available on the free tier.
Visual anomalies: If your telemetry includes dashboards, heatmaps, or screenshots, vision models such as Kimi VL A3B and Gemma 3 27B can detect visual regressions directly.
Embedding baselines: BGE-Large and E5-Large create semantic baselines for clustering and outlier detection without consuming chat model quota.

Operational Best Practices

Stream responses for real-time dashboards. Oxlo.ai supports streaming responses, so you can render partial analysis while the model completes its reasoning.

Use function calling for closed-loop remediation. Define tools that create tickets, restart services, or page on-call staff. The model can decide whether an anomaly warrants immediate action or further investigation.

Combine vision and text. For infrastructure that generates charts or screenshots, pass images to a vision-capable model via the chat completions endpoint. Oxlo.ai supports vision input for compatible models.

Maintain state with multi-turn conversations. Interactive debugging sessions benefit from retaining context across turns. Because Oxlo.ai does not charge by the token, extending the conversation history does not inflate inference costs.

Pricing and Getting Started

Oxlo.ai offers a free tier with 60 requests per day, access to 16+ free models, and a 7-day full-access trial. DeepSeek V3.2 is included in the free tier, making it straightforward to prototype detection pipelines with no initial cost. Paid plans provide 1,000 requests per day on Pro and 5,000 requests per day on Premium, with priority queue access and all models unlocked. For enterprise workloads, dedicated GPUs and custom pricing are available.

Exact request costs depend on your plan. See the Oxlo.ai pricing page for current rates. Remember that request-based pricing means a 500-line log dump costs the same as a ten-word prompt. For anomaly detection workloads that are inherently long-context, that predictability is a structural advantage over token-based billing.

Building Multimodal Models with LLMs: A Step-by-Step Guide

shashank ms — Mon, 27 Jul 2026 07:33:46 +0000

We are building a multimodal diagnostic agent that ingests screenshots or technical diagrams and returns structured bug reports. This gives QA teams and frontend engineers a fast way to triage visual issues without maintaining custom computer vision pipelines. The entire agent runs against Oxlo.ai's vision-capable API using the standard OpenAI SDK.

What you'll need

Python 3.10 or newer
The OpenAI SDK: pip install openai
An Oxlo.ai API key from https://portal.oxlo.ai
A sample image file such as a PNG screenshot

Step 1: Configure the Oxlo.ai client

Create a client instance pointed at Oxlo.ai. We will use kimi-k2.6 because it handles vision, reasoning, and long context in a single request.

from openai import OpenAI
import base64

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key="YOUR_OXLO_API_KEY"  # from https://portal.oxlo.ai
)

Step 2: Prepare image input for the API

Oxlo.ai accepts images as base64 data URLs inside the chat messages payload. This helper reads a local file and returns the encoded string.

def encode_image(image_path: str) -> str:
    with open(image_path, "rb") as image_file:
        return base64.b64encode(image_file.read()).decode("utf-8")

image_b64 = encode_image("dashboard_bug.png")
data_url = f"data:image/png;base64,{image_b64}"

Step 3: Define the system prompt

A strong system prompt keeps the model's output structured and actionable. Store it as a constant so you can iterate without touching the request logic.

SYSTEM_PROMPT = """You are a visual QA engineer. Analyze the provided screenshot and identify UI bugs, layout issues, or accessibility problems.

Return your findings as a JSON object with this exact structure:
{
  "issues": [
    {
      "severity": "critical|major|minor",
      "component": "name of UI element",
      "description": "what is wrong",
      "recommendation": "how to fix it"
    }
  ],
  "summary": "one sentence overall assessment"
}

Be concise. If no issues are found, return an empty issues array."""

Step 4: Build the analyzer function

Combine the text prompt and the image into a single user message, then send it to Oxlo.ai. The OpenAI-compatible format lets you mix text and image_url content blocks seamlessly.

def analyze_screenshot(image_path: str, question: str = "Analyze this screenshot for UI issues.") -> str:
    b64 = encode_image(image_path)
    url = f"data:image/png;base64,{b64}"
    
    response = client.chat.completions.create(
        model="kimi-k2.6",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {
                "role": "user",
                "content": [
                    {"type": "text", "text": question},
                    {"type": "image_url", "image_url": {"url": url}},
                ],
            },
        ],
        max_tokens=1024,
    )
    return response.choices[0].message.content

Run it

Call the analyzer on a real image and print the result. If you do not have a dashboard bug handy, any screenshot with text and buttons will produce useful output.

if __name__ == "__main__":
    report = analyze_screenshot("dashboard_bug.png")
    print(report)

Example output from a malformed navigation bar:

{
  "issues": [
    {
      "severity": "major",
      "component": "navigation sidebar",
      "description": "The sidebar overlaps the main content area at viewport widths below 768px.",
      "recommendation": "Add a media query to collapse the sidebar into a hamburger menu on mobile breakpoints."
    },
    {
      "severity": "minor",
      "component": "submit button",
      "description": "Color contrast ratio is 2.9:1, below WCAG AA standards.",
      "recommendation": "Darken the button background to #005fcc to reach a 4.5:1 ratio."
    }
  ],
  "summary": "Two layout and accessibility issues detected that should be fixed before release."
}

Next steps

Batch process an entire directory of screenshots by wrapping analyze_screenshot in a loop and writing the JSON results to a log file. You can also add Oxlo.ai function calling to the system prompt so the agent opens Jira tickets automatically when it detects critical severity issues.

Unlocking Audio Analysis with LLMs

shashank ms — Mon, 27 Jul 2026 07:32:28 +0000

Audio is the fastest-growing unstructured data type in enterprise stacks, yet it remains the hardest to query. Unlike text logs or JSON events, you cannot grep a podcast or run SQL over a support call. The practical pattern is to transcribe speech to text with an automatic speech recognition model, then apply a large language model to the transcript for summarization, sentiment analysis, entity extraction, or compliance scoring. Oxlo.ai hosts both the transcription and reasoning layers under a single request-based pricing model, removing the cost uncertainty that scales with transcript length on token-based providers.

The Transcription-Reasoning Pipeline

Most audio analysis workloads follow a two-stage pipeline. First, an audio transcription model converts speech into text. Second, an LLM reasons over that text. On Oxlo.ai, both stages use the same OpenAI-compatible SDK and base URL, so you do not need separate clients or authentication schemes.

For transcription, Oxlo.ai offers Whisper Large v3, Whisper Turbo, and Whisper Medium through the audio/transcriptions endpoint. For reasoning, you can route the resulting text to any chat model, such as Llama 3.3 70B for general analysis, Qwen 3 32B for multilingual agent workflows, or DeepSeek R1 671B MoE for deep reasoning over complex technical conversations.

Code Example: Analyzing a Support Call

The following Python example uses the OpenAI SDK to transcribe a support call and then extract structured insights with an LLM. Because Oxlo.ai is fully OpenAI SDK compatible, the only change is the base_url.

import openai
import json

client = openai.OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key="YOUR_OXLO_API_KEY"
)

# Stage 1: Transcribe audio to text
with open("support_call.mp3", "rb") as audio_file:
    transcript = client.audio.transcriptions.create(
        model="whisper-large-v3",
        file=audio_file,
        response_format="text"
    )

# Stage 2: Structured analysis with an LLM
completion = client.chat.completions.create(
    model="llama-3.3-70b",
    messages=[
        {
            "role": "system",
            "content": (
                "You are a call-quality analyst. Read the transcript and produce JSON with "
                "keys: sentiment (positive, neutral, negative), speaker_roles (list), and "
                "action_items (list)."
            )
        },
        {"role": "user", "content": transcript}
    ],
    response_format={"type": "json_object"}
)

result = json.loads(completion.choices[0].message.content)
print(json.dumps(result, indent=2))

The same pattern works for batch processing. Point the transcription step at an S3 bucket or local directory, then fan out the text to the chat endpoint with asynchronous workers. Because Oxlo.ai has no cold starts on popular models, the first request in a batch returns as quickly as the hundredth.

Why Request Pricing Matters for Audio

Audio produces verbose transcripts. A thirty-minute interview can easily generate thousands of tokens of text. On token-based providers, the cost to analyze that transcript scales linearly with its length, and if you run multiple analysis passes, the multiplier compounds.

Oxlo.ai uses request-based pricing: one flat cost per API request regardless of prompt length. That means analyzing a long transcript costs the same as a short one, and running ten reasoning passes over the same text costs exactly ten requests. For long-context and agentic audio workflows, this can be 10-100x cheaper than token-based alternatives. See the exact tiers on the Oxlo.ai pricing page.

Scaling Beyond Single Files

Production audio pipelines rarely stop at one file. You can build stateful agents by combining transcription with function calling. For example, after transcribing a customer call, an LLM can decide whether to open a ticket in your CRM, escalate to a human, or append notes to an existing account. Oxlo.ai supports function calling and multi-turn conversations on its chat models, so the decision logic stays in the same request-based layer.

If you need to preserve context across long recordings, use a model with a large context window such as Kimi K2.6, which offers a 131K context and advanced reasoning, or DeepSeek V4 Flash with its 1M context window for near state-of-the-art open-source reasoning over entire audiobooks or day-long meeting archives.

Models and Formats on Oxlo.ai

Oxlo.ai provides dedicated endpoints for every stage of an audio analysis pipeline.

Speech-to-text: Whisper Large v3, Whisper Turbo, and Whisper Medium via audio/transcriptions.
Text-to-speech: Kokoro 82M via audio/speech for building voice agents that respond after analysis.
Reasoning: Llama 3.3 70B, Qwen 3 32B, DeepSeek R1 671B MoE, Kimi K2.6, GLM 5, and others via chat/completions.
Structured output: JSON mode and function calling are supported on compatible chat models.

All endpoints share the same base URL and authentication, so switching from transcription to reasoning is a single line change in your client configuration.

Next Steps

If you are currently paying token-based rates to analyze transcripts, moving the workload to Oxlo.ai is a drop-in replacement. The Free plan includes 60 requests per day and a 7-day full-access trial, which is enough to prototype a full transcription-to-insights pipeline without a credit card. When you are ready to scale, paid plans add daily request allotments, priority queue access, and dedicated GPU options. Review the details at https://oxlo.ai/pricing and point your OpenAI client to https://api.oxlo.ai/v1 to start.</

Integrating LLMs with Virtual Reality: A Step-by-Step Guide

shashank ms — Mon, 27 Jul 2026 05:38:11 +0000

We are going to build a VR world generator that turns text prompts into interactive A-Frame scenes. A FastAPI backend calls Oxlo.ai to generate 3D markup and atmospheric narration, while a lightweight frontend renders it in the browser. This is useful for rapid prototyping of virtual environments, interactive storytelling, or any application where non-developers need to create spatial experiences.

What you'll need

Before starting, make sure you have Python 3.10 or newer installed. You will also need an Oxlo.ai API key from https://portal.oxlo.ai and the OpenAI SDK. Oxlo.ai uses request-based pricing, so sending long scene descriptions or multi-turn context does not inflate your cost the way token-based billing would. For details, see https://oxlo.ai/pricing.

Python 3.10+
pip install openai fastapi uvicorn
An Oxlo.ai API key
A modern web browser with WebGL support

1. Set up the Oxlo.ai client

First, I verify that I can reach Oxlo.ai and get coherent scene descriptions. I create vr_builder.py and initialize the OpenAI-compatible client pointing at Oxlo.ai.

from openai import OpenAI

client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")

user_message = "A bioluminescent cave with glowing crystals and still water."

response = client.chat.completions.create(
    model="llama-3.3-70b",
    messages=[
        {"role": "system", "content": "You are a VR scene designer. Respond with a one-sentence description of the environment."},
        {"role": "user", "content": user_message},
    ],
)

print(response.choices[0].message.content)

2. Define the system prompt and scene generator

Now I lock down the output format. The model must return valid JSON containing the A-Frame scene markup, a narration string, and a list of interactive object IDs. I define the system prompt as a constant so I can tweak it without touching the rest of the logic.

SYSTEM_PROMPT = """You are a VR world builder. Given a user description, generate a JSON object with exactly three keys:
- "scene_html": A string containing a complete A-Frame <a-scene> element. Include a sky, ground plane, lighting, and at least three interactive objects. Every interactive object must have a unique id and class="clickable". Use only standard A-Frame primitives.
- "narration": A short atmospheric narration to display to the user.
- "objects": A list of strings containing the id of every interactive object.
Output only valid JSON. Do not wrap the output in markdown."""

Then I wire it into a reusable function that calls Oxlo.ai and parses the result.

import json
from openai import OpenAI

client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")

def generate_vr_scene(user_message: str) -> dict:
    response = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": user_message},
        ],
    )
    content = response.choices[0].message.content
    return json.loads(content)

3. Build the FastAPI backend

Next, I wrap the generator in a FastAPI application. The /generate endpoint accepts a prompt, calls Oxlo.ai, and returns the parsed scene data. I add CORS so the frontend can reach it during local development.

import json
from openai import OpenAI
from fastapi import FastAPI
from fastapi.middleware.cors import CORSMiddleware
from pydantic import BaseModel
import uvicorn

client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")

SYSTEM_PROMPT = """You are a VR world builder. Given a user description, generate a JSON object with exactly three keys:
- "scene_html": A string containing a complete A-Frame <a-scene> element. Include a sky, ground plane, lighting, and at least three interactive objects. Every interactive object must have a unique id and class="clickable". Use only standard A-Frame primitives.
- "narration": A short atmospheric narration to display to the user.
- "objects": A list of strings containing the id of every interactive object.
Output only valid JSON. Do not wrap the output in markdown."""

app = FastAPI()

app.add_middleware(
    CORSMiddleware,
    allow_origins=["*"],
    allow_methods=["*"],
    allow_headers=["*"],
)

class PromptRequest(BaseModel):
    prompt: str

def generate_vr_scene(user_message: str) -> dict:
    response = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": user_message},
        ],
    )
    content = response.choices[0].message.content
    return json.loads(content)

@app.post("/generate")
def create_scene(req: PromptRequest):
    return generate_vr_scene(req.prompt)

if __name__ == "__main__":
    uvicorn.run(app, host="0.0.0.0", port=8000)

4. Create the A-Frame frontend

The frontend is a single HTML file that hosts an iframe for the VR scene. It posts the user prompt to the FastAPI backend, then injects the returned HTML into the iframe using srcdoc. This keeps the generated world isolated and avoids A-Frame DOM merge issues.

<!DOCTYPE html>
<html>
<head>
  <style>
    body { margin: 0; font-family: sans-serif; display: flex; flex-direction: column; height: 100vh; background: #0b0c10; color: #c5c6c7; }
    #controls { padding: 1rem; background: #1f2833; border-bottom: 1px solid #45a29e; }
    input, button { padding: 0.5rem; font-size: 1rem; margin-right: 0.5rem; }
    #vr-frame { flex: 1; border: none; }
    #narration { margin-top: 0.5rem; font-style: italic; }
  </style>
</head>
<body>
  <div id="controls">
    <input id="prompt" type="text" value="A cyberpunk alley with neon signs and rain puddles" size="50">
    <button onclick="generate()">Generate World</button>
    <div id="narration">Ready.</div>
  </div>
  <iframe id="vr-frame" allow="vr"></iframe>
  <script>
    async function generate() {
      const user_message = document.getElementById('prompt').value;
      const res = await fetch('http://localhost:8000/generate', {
        method: 'POST',
        headers: {'Content-Type': 'application/json'},
        body: JSON.stringify({prompt: user_message})
      });
      const data = await res.json();
      document.getElementById('narration').innerText = data.narration;

      const doc = `<!DOCTYPE html>
<html>
<head><script src="https://aframe.io/releases/1.6.0/aframe.min.js"></script></head>
<body style="margin:0">${data.scene_html}</body>
</html>`;
      document.getElementById('vr-frame').srcdoc = doc;
      window.currentScene = data;
    }
  </script>
</body>
</html>

5. Add dynamic object interactivity

Static scenes are only half the story. I want users to click an object and receive AI-generated lore. I add an /interact endpoint that uses qwen-3-32b for creative reasoning, then expose buttons for each object. When a button is clicked, the frontend manipulates the iframe DOM directly to provide immediate visual feedback.

Add the new endpoint to vr_builder.py before the uvicorn block.

class InteractRequest(BaseModel):
    object_id: str
    scene_context: str

@app.post("/interact")
def interact(req: InteractRequest):
    system_prompt = "You are a VR lore master. Given an object ID and scene context, return a JSON object with 'lore' (one mysterious sentence) and 'effect' (one of: glow, pulse, shrink). Output only JSON."
    user_message = f"Object: {req.object_id}\nContext: {req.scene_context}"

    response = client.chat.completions.create(
        model="qwen-3-32b",
        messages=[
            {"role": "system", "content": system_prompt},
            {"role": "user", "content": user_message},
        ],
    )
    return json.loads(response.choices[0].message.content)

Then extend the frontend to render interaction buttons after each generation.

    async function generate() {
      const user_message = document.getElementById('prompt').value;
      const res = await fetch('http://localhost:8000/generate', {
        method: 'POST',
        headers: {'Content-Type': 'application/json'},
        body: JSON.stringify({prompt: user_message})
      });
      const data = await res.json();
      document.getElementById('narration').innerText = data.narration;

      const doc = `<!DOCTYPE html>
<html>
<head><script src="https://aframe.io/releases/1.6.0/aframe.min.js"></script></head>
<body style="margin:0">${data.scene_html}</body>
</html>`;
      document.getElementById('vr-frame').srcdoc = doc;

      const controls = document.getElementById('controls');
      controls.querySelectorAll('.obj-btn').forEach(b => b.remove());
      data.objects.forEach(obj => {
        const btn = document.createElement('button');
        btn.className = 'obj-btn';
        btn.innerText = obj;
        btn.onclick = async () => {
          const ires = await fetch('http://localhost:8000/interact', {
            method: 'POST',
            headers: {'Content-Type': 'application/json'},
            body: JSON.stringify({object_id: obj, scene_context: data.narration})
          });
          const idata = await ires.json();
          document.getElementById('narration').innerText = idata.lore;
          const el = document.getElementById('vr-frame').contentDocument.getElementById(obj);
          if (el && idata.effect === 'glow') {
            el.setAttribute('material', 'color', '#ffaa00');
          }
        };
        controls.appendChild(btn);
      });
    }

Run it

Start the backend from your terminal.

python vr_builder.py

Open index.html in your browser, or serve it with python -m http.server 8080 and visit http://localhost:8080. Enter a prompt and click Generate World. You should see an A-Frame scene load in the iframe along with a narration line. Example output after clicking Generate World:

{
  "scene_html": "<a-scene>\n  <a-sky color=\"#050510\"></a-sky>\n  <a-plane position=\"0 0 -4\" rotation=\"-90 0 0\" width=\"20\" height=\"20\" color=\"#222\"></a-plane>\n  <a-box id=\"crate\" class=\"clickable\" position=\"-1 0.5 -3\" color=\"#654321\"></a-box>\n  <a-sphere id=\"orb\" class=\"clickable\" position=\"1 1 -3\" color=\"#00ffcc\"></a-sphere>\n  <a-cylinder id=\"pillar\" class=\"clickable\" position=\"0 1 -4\" radius=\"0.3\" height=\"2\" color=\"#777\"></a-cylinder>\n  <a-light type=\"ambient\" color=\"#444\"></a-light>\n  <a-light type=\"point\" position=\"2 3 2\" color=\"#ffaa00\"></a-light>\n</a-scene>",
  "narration": "The alley hums with hidden current. Three artifacts await your attention.",
  "objects": ["crate", "orb", "pillar"]
}

Clicking the orb button might return:

{
  "lore": "The orb pulses with a frequency that predates the city above.",
  "effect": "glow"
}

Next steps

Swap the text buttons for speech input by piping microphone audio through Oxlo.ai's Whisper endpoint, then synthesize responses with Kokoro TTS for a fully hands-free VR interface. If you want the world to remember state across sessions, store object embeddings using Oxlo.ai's BGE-Large endpoint and retrieve them on each load to build persistent narrative memory.

Advancing Video Analysis with LLMs

shashank ms — Mon, 27 Jul 2026 05:35:55 +0000

Video is the fastest-growing data modality, yet extracting structured insight from it remains a hard engineering problem. Traditional computer vision pipelines rely on specialized models for detection, classification, and tracking. Large language models with vision capabilities now offer a unified alternative. They can consume sampled frames, detect temporal events, and generate structured narratives from raw video. The bottleneck is no longer model capability. It is context length and inference cost when you submit dozens or hundreds of frames.

The Challenge of Video as a Sequence

A video is not a single image. It is a time-ordered sequence with motion, audio, and narrative structure. Feeding every frame into a model is computationally impossible, so engineers must sample, compress, or summarize. The core challenge is preserving temporal coherence while staying within context window limits and budget constraints. A one-minute clip at 24 frames per second contains 1,440 frames. Even aggressive subsampling to one frame per second yields 60 images. When each image is worth hundreds or thousands of tokens, the prompt size grows rapidly.

Architectural Patterns for LLM Video Analysis

Most production systems use one of three patterns.

Uniform frame sampling. Extract frames at fixed intervals, encode them as base64 JPEGs, and pass them to a vision-capable chat model. This is simple and works well for static scenes or slide-based content.

Scene segmentation. Compute perceptual hashes or frame differences to detect shot boundaries. Only keyframes representing new scenes are sent to the model. This reduces redundancy and keeps the context window focused.

Hierarchical analysis. A lightweight model or embedding first clusters frames into scenes. Each scene is summarized independently by a strong vision LLM. A second LLM pass then synthesizes the scene summaries into a coherent timeline. This pattern mimics human skim-reading and scales to hours of footage.

Each pattern benefits from models that support long contexts and vision. Oxlo.ai hosts vision-capable models including Kimi K2.6 with 131K context, Gemma 3 27B, and the Qwen 3 family, all accessible through a single OpenAI-compatible endpoint.

Multimodal Models and Long Context

Modern vision-language models can reason across many images in a single conversation turn. Kimi K2.6 supports advanced reasoning, agentic coding, and vision with a 131K context window. Gemma 3 27B offers strong vision performance. Qwen 3 32B and GLM 5 handle multilingual and long-horizon agentic tasks.

The problem is economic, not architectural. Under token-based pricing, every image in the prompt adds to the input token count. A 20-frame request can already exceed the cost of the output generation. Under Oxlo.ai request-based pricing, the cost is flat per API request regardless of prompt length. Unlike token-based providers such as Together AI, Fireworks AI, OpenRouter, Replicate, and Anyscale, Oxlo.ai does not penalize you for sending more frames. For video analysis, where long context is the norm, this structural difference means predictable costs. Oxlo.ai also delivers no cold starts on popular models, which keeps batch inference latency stable.

Implementing Frame-Level Analysis

The following Python script uses OpenCV to sample frames and the OpenAI SDK to send them to Oxlo.ai. Because Oxlo.ai is fully OpenAI SDK compatible, you only need to change the base_url.

import os
import cv2
import base64
from openai import OpenAI

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key=os.environ["OXLO_API_KEY"]
)

def sample_frames(video_path: str, interval_seconds: int = 5):
    cap = cv2.VideoCapture(video_path)
    fps = cap.get(cv2.CAP_PROP_FPS)
    interval = int(fps * interval_seconds)
    frames = []
    count = 0

    while cap.isOpened():
        ret, frame = cap.read()
        if not ret:
            break
        if count % interval == 0:
            success, buffer = cv2.imencode(".jpg", frame)
            if success:
                b64 = base64.b64encode(buffer).decode("utf-8")
                frames.append(f"data:image/jpeg;base64,{b64}")
        count += 1

    cap.release()
    return frames

frames = sample_frames("demo.mp4", interval_seconds=10)

# Build a multimodal message with up to 8 frames
content = [{"type": "text", "text": (
    "Analyze the video and return a JSON object with keys: "
    "summary, key_events, and visible_objects."
)}]
content += [
    {"type": "image_url", "image_url": {"url": url}}
    for url in frames[:8]
]

response = client.chat.completions.create(
    model="kimi-k2.6",  # vision-capable, 131K context window
    messages=[{"role": "user", "content": content}],
    response_format={"type": "json_object"}
)

print(response.choices[0].message.content)

This example uses JSON mode to enforce structured output. You can extend it with function calling to populate a database or trigger downstream workflows.

Agentic Workflows for Temporal Reasoning

Single-pass analysis often misses causality. A model may see a person pick up an object in frame 10 and set it down in frame 30, but without explicit reasoning it may fail to track state changes across the sequence.

Agentic workflows solve this by treating video analysis as a multi-turn conversation. The first request generates a scene-level description. A second request asks follow-up questions. Function calling lets the model query external tools, such as a vector store of object embeddings or a taxonomy service.

Oxlo.ai supports streaming responses, function calling, and multi-turn conversations out of the box. Models like Qwen 3 32B, GLM 5, and DeepSeek R1 671B are particularly strong at agentic reasoning and complex tool use. Because each turn can carry the full video context, token-based billing would compound costs quickly. With Oxlo.ai request-based pricing, iterative refinement stays economically viable.

Cost Predictability at Scale

Token-based providers bill by input and output tokens. A single 1080p frame encoded as a base64 image URL can represent thousands of tokens. When you send 30, 50, or 100 frames, the input token count dominates the bill.

Oxlo.ai request-based pricing removes this variable. One flat cost per API request means your budget scales with the number of videos analyzed, not the number of frames or tokens in each prompt. For production pipelines processing thousands of clips, Oxlo.ai request-based pricing can be 10-100x cheaper than token-based alternatives for long-context workloads. There are no cold starts, and latency remains consistent. See https://oxlo.ai/pricing for current plan details.

Conclusion

Video analysis with LLMs is moving from experiment to production infrastructure. The deciding factor is no longer model capability alone. It is the economics of context. Oxlo.ai offers 45+ open-source and proprietary models, vision support, full OpenAI SDK compatibility, and request-based pricing that removes the penalty for long inputs. For teams building the next generation of video understanding products, that combination is a practical foundation.

Unlocking Image Captioning with LLMs

shashank ms — Mon, 27 Jul 2026 05:33:27 +0000

Image captioning sits at the intersection of computer vision and language understanding. Feeding an image to a large language model and receiving a coherent, context-aware description is now a production requirement for content platforms, accessibility tools, and asset management pipelines. The challenge is no longer whether an LLM can describe an image, but how to do so at scale without costs that balloon with every pixel or token.

Why Image Captioning Needs More Than a Vanilla LLM

A text-only LLM cannot see. Production pipelines need a vision-language model that ingests images alongside prompts, handles high-resolution inputs, and returns structured data. Reliability matters: low-latency streaming, JSON mode for schema enforcement, and multi-turn conversation support let you build captioning systems that integrate cleanly into existing workflows. Cold starts break batch pipelines, so consistent inference latency is critical.

Vision Models Available on Oxlo.ai

Oxlo.ai offers vision-capable models through a single OpenAI-compatible endpoint. Gemma 3 27B delivers strong performance for general image captioning, while Kimi VL A3B provides efficient multimodal reasoning. If your pipeline combines vision with long-document context or agentic steps, Kimi K2.6 supports 131K tokens, advanced reasoning, and vision inputs. Switching between these models requires only changing the model parameter in your request.

Implementation with the OpenAI SDK

Because Oxlo.ai is fully OpenAI SDK compatible, you can use the same Python, Node.js, or cURL patterns you already know. The only change is the base URL.

import os
import base64
from openai import OpenAI

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key=os.environ["OXLO_API_KEY"]
)

def encode_image(path):
    with open(path, "rb") as f:
        return base64.b64encode(f.read()).decode("utf-8")

image_b64 = encode_image("photo.jpg")

response = client.chat.completions.create(
    model="gemma-3-27b",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "Generate a concise, neutral caption describing the main subject and setting of this image."
                },
                {
                    "type": "image_url",
                    "image_url": {"url": f"data:image/jpeg;base64,{image_b64}"}
                }
            ]
        }
    ],
    max_tokens=256
)

caption = response.choices[0].message.content
print(caption)

Scaling Batch Workloads with Request-Based Pricing

On token-based platforms, images are converted into large token sequences before the prompt text is even counted. Captioning a high-resolution archive or a video frame sequence causes costs to grow unpredictably with input size. Oxlo.ai uses flat, request-based pricing: one fixed cost per API call regardless of image resolution or prompt length. For long-context and agentic workloads, this model can be significantly cheaper than token-based billing from providers such as Together AI, Fireworks AI, OpenRouter, Replicate, and Anyscale. You can verify current plans on the Oxlo.ai pricing page.

Enforcing Structure with JSON Mode

Raw text captions are hard to parse. Oxlo.ai supports JSON mode and function calling, so you can enforce schemas directly in the chat completions endpoint.

import os
import base64
import json
from openai import OpenAI

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key=os.environ["OXLO_API_KEY"]
)

image_b64 = base64.b64encode(open("photo.jpg", "rb").read()).decode("utf-8")

response = client.chat.completions.create(
    model="gemma-3-27b",
    messages=[
        {
            "role": "system",
            "content": "You are a captioning assistant. Respond with valid JSON containing keys: caption, tags, setting."
        },
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Describe this image."},
                {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{image_b64}"}}
            ]
        }
    ],
    response_format={"type": "json_object"},
    max_tokens=512
)

structured = json.loads(response.choices[0].message.content)
print(structured)

From Prototype to Production

Oxlo.ai removes friction from the prototype stage through its free tier, which includes 60 requests per day and access to 16+ models, plus a 7-day full-access trial. Popular models have no cold starts, so batch jobs and user-facing apps get predictable latency. When volume grows, the Pro and Premium plans offer dedicated request pools and priority queue access. Because the platform is a drop-in replacement for the OpenAI SDK, migrating an existing captioning service takes minutes, not days.

Optimizing LLM Inference for Dialogue Systems: Best Practices and Considerations

shashank ms — Mon, 27 Jul 2026 03:36:23 +0000

We are building a stateful customer support agent that handles order lookups, refund requests, and escalation across multi-turn conversations. It is aimed at engineering teams who need to automate tier-1 support without losing context or ballooning inference costs. I will walk through the exact code I shipped, using Oxlo.ai for the LLM layer.

What you'll need

Python 3.10 or newer
The OpenAI SDK: pip install openai
An Oxlo.ai API key from https://portal.oxlo.ai

Step 1: Verify connectivity to Oxlo.ai

I always start with a smoke test to confirm the endpoint and key are healthy before adding any logic.

from openai import OpenAI
import os

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key=os.environ.get("OXLO_API_KEY", "YOUR_OXLO_API_KEY")
)

response = client.chat.completions.create(
    model="llama-3.3-70b",
    messages=[{"role": "user", "content": "ping"}],
)

print("Endpoint ready:", response.choices[0].message.content[:20])

Step 2: Lock down the system prompt

The system prompt is the contract that keeps the agent from drifting. I keep it in a top-level constant so product teams can edit it without touching the router.

SYSTEM_PROMPT = """You are a tier-1 support agent for Acme Gadgets.
You help customers with order tracking, refunds, and basic troubleshooting.
Guidelines:
- Always ask for an order ID before looking up details.
- If a customer asks for a refund, check order status first.
- Never invent order data. Use the lookup_order tool.
- Keep responses concise, max two sentences.
- If the user mentions fire or safety hazards, reply 'ESCALATE-HUMAN'."""

Step 3: Build a minimal session manager

Dialogue lives or dies by state. This class appends user and assistant messages so the model sees the full thread on every turn.

class SupportSession:
    def __init__(self):
        self.messages = [{"role": "system", "content": SYSTEM_PROMPT}]

    def user_says(self, text):
        self.messages.append({"role": "user", "content": text})

    def assistant_says(self, text):
        self.messages.append({"role": "assistant", "content": text})
        return text

Step 4: Define the tool schema and a stub backend

I expose one tool for order lookups. In production this hits your database, but a hardcoded dict is enough to verify the loop.

import json

TOOLS = [
    {
        "type": "function",
        "function": {
            "name": "lookup_order",
            "description": "Retrieve current status of a customer order",
            "parameters": {
                "type": "object",
                "properties": {
                    "order_id": {
                        "type": "string",
                        "description": "Order identifier, e.g. ORD-12345"
                    }
                },
                "required": ["order_id"]
            }
        }
    }
]

def lookup_order(order_id: str):
    db = {
        "ORD-12345": {"status": "delivered", "item": "Mechanical Keyboard", "refund_eligible": True},
        "ORD-67890": {"status": "in_transit", "item": "USB-C Hub", "refund_eligible": False}
    }
    return db.get(order_id, {"status": "not_found"})

Step 5: Wire up the turn handler with automatic tool execution

This is the core router. It sends the conversation to Oxlo.ai, detects if the model wants to call a tool, executes the function locally, and sends the result back for a final natural-language response.

def handle_turn(client, session, user_text):
    session.user_says(user_text)

    response = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=session.messages,
        tools=TOOLS,
        tool_choice="auto",
    )

    msg = response.choices[0].message

    if msg.tool_calls:
        session.messages.append(msg)

        for tc in msg.tool_calls:
            if tc.function.name == "lookup_order":
                args = json.loads(tc.function.arguments)
                result = lookup_order(args["order_id"])
                session.messages.append({
                    "role": "tool",
                    "tool_call_id": tc.id,
                    "content": json.dumps(result)
                })

        follow_up = client.chat.completions.create(
            model="llama-3.3-70b",
            messages=session.messages,
        )
        reply = follow_up.choices[0].message.content
        return session.assistant_says(reply)

    return session.assistant_says(msg.content)

Step 6: Compress context when threads get long

After a dozen turns the thread gets noisy. Because Oxlo.ai charges per request instead of per token, I can afford to run a summarization pass without sweating the length. This keeps the model focused while capping the message list.

def maybe_compress(session, client):
    if len(session.messages) > 14:
        history = json.dumps(session.messages[1:-6])
        summary_resp = client.chat.completions.create(
            model="llama-3.3-70b",
            messages=[
                {"role": "system", "content": "Summarize this support chat for an agent. Keep facts, drop fluff."},
                {"role": "user", "content": history}
            ],
        )
        summary = summary_resp.choices[0].message.content
        session.messages = [
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": f"Previous conversation summary: {summary}"}
        ] + session.messages[-6:]

Run it

Here is how I test the agent locally. I inject the compression check between turns to simulate a long session.

session = SupportSession()

print("Bot:", handle_turn(client, session, "Hi, where is my order? I think the ID is ORD-12345"))
maybe_compress(session, client)
print("Bot:", handle_turn(client, session, "Can I get a refund then?"))

Example output:

Bot: I found your order. It shows as delivered.
Bot: Yes, that order is eligible for a refund. I can start that process for you now.

Next steps

Swap in kimi-k2.6 for the summarization step if you want stronger reasoning over long context, or try deepseek-v3.2 on the free tier to prototype without spending quota. If you move to production, replace the lookup_order stub with an async call to your orders API so the tool handler does not block the event loop.

Integrating LLM with Augmented Reality Applications: A Step-by-Step Guide

shashank ms — Mon, 27 Jul 2026 03:34:43 +0000

We are building a web-based AR repair assistant that takes a plain-text description of a physical scene, asks an LLM for contextual annotations, and renders those instructions as overlays on a live camera feed. It helps developers prototype intelligent AR guidance without native app stores or unpredictable token bills. Oxlo.ai fits here because its request-based pricing means a long, detailed scene description costs the same flat rate as a short one.

What you'll need

Python 3.10 or newer
An Oxlo.ai API key from https://portal.oxlo.ai
The OpenAI SDK and FastAPI: pip install openai fastapi uvicorn python-multipart
A modern browser with camera access

Step 1: Scaffold the backend

Create a new project folder and a main.py file. I will set up a FastAPI app with CORS enabled so the frontend can reach it during local development.

from fastapi import FastAPI
from fastapi.middleware.cors import CORSMiddleware
from pydantic import BaseModel
from openai import OpenAI
import json

app = FastAPI()

app.add_middleware(
    CORSMiddleware,
    allow_origins=["*"],
    allow_methods=["*"],
    allow_headers=["*"],
)

class SceneRequest(BaseModel):
    description: str
    task: str = "diagnose"

@app.get("/health")
def health():
    return {"status": "ok"}

Step 2: Write the system prompt

The system prompt tells the model how to format AR overlays. I want valid JSON with label text and normalized screen coordinates so the frontend can position divs directly.

SYSTEM_PROMPT = """You are an AR repair assistant. The user describes what they see in their physical environment.

Respond with a JSON object containing:
- "thought": a brief analysis of the scene
- "overlays": a list of objects, each with:
  - "label": the text to display
  - "x": horizontal position from 0.0 (left) to 1.0 (right)
  - "y": vertical position from 0.0 (top) to 1.0 (bottom)
  - "color": a CSS color for the label

Keep labels under 60 characters. Position overlays near the relevant object described by the user. If the user describes multiple items, prioritize the one most relevant to the task."""

Step 3: Connect to Oxlo.ai and create the annotation endpoint

Here I wire the endpoint to Oxlo.ai. I use the OpenAI SDK as a drop-in client and call Llama 3.3 70B. Because Oxlo.ai uses request-based pricing, not token-based pricing, a long scene description does not increase the cost.

client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")

@app.post("/annotate")
def annotate_scene(req: SceneRequest):
    user_message = f"Task: {req.task}\nScene: {req.description}"

    response = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": user_message},
        ],
    )

    raw = response.choices[0].message.content
    # Strip markdown fences if the model wraps JSON
    cleaned = raw.strip()
    if cleaned.startswith("

```"):
        cleaned = cleaned.split("```

")[1]
        if cleaned.startswith("json"):
            cleaned = cleaned[4:]
        cleaned = cleaned.strip()

    return json.loads(cleaned)

Step 4: Build the AR frontend

Create a static/ folder and add index.html. The page requests camera access, shows the video full-screen, and provides a text box for the user to describe what they see.

<!DOCTYPE html>
<html>
<head>
  <meta charset="UTF-8">
  <meta name="viewport" content="width=device-width, initial-scale=1.0">
  <title>AR LLM Overlay</title>
  <style>
    body { margin: 0; overflow: hidden; background: #000; font-family: system-ui, sans-serif; }
    #video { position: fixed; top: 0; left: 0; width: 100vw; height: 100vh; object-fit: cover; }
    #ui { position: fixed; bottom: 20px; left: 20px; right: 20px; z-index: 10; display: flex; gap: 10px; }
    input { flex: 1; padding: 12px; border: none; border-radius: 8px; font-size: 16px; }
    button { padding: 12px 20px; border: none; border-radius: 8px; background: #0af; color: #fff; font-weight: 600; }
    .overlay { position: fixed; padding: 6px 10px; border-radius: 6px; background: rgba(0,0,0,0.6); color: #fff; font-size: 14px; pointer-events: none; transform: translate(-50%, -50%); border: 2px solid; }
  </style>
</head>
<body>
  <video id="video" autoplay playsinline></video>
  <div id="ui">
    <input id="desc" placeholder="Describe what you see..." />
    <button onclick="analyze()">Analyze</button>
  </div>
  <div id="overlays"></div>

  <script>
    const video = document.getElementById('video');
    navigator.mediaDevices.getUserMedia({ video: { facingMode: 'environment' } })
      .then(stream => video.srcObject = stream);

    async function analyze() {
      const desc = document.getElementById('desc').value;
      const res = await fetch('http://localhost:8000/annotate', {
        method: 'POST',
        headers: { 'Content-Type': 'application/json' },
        body: JSON.stringify({ description: desc, task: 'diagnose' })
      });
      const data = await res.json();
      render(data.overlays);
    }

    function render(overlays) {
      const container = document.getElementById('overlays');
      container.innerHTML = '';
      overlays.forEach(o => {
        const el = document.createElement('div');
        el.className = 'overlay';
        el.textContent = o.label;
        el.style.left = (o.x * 100) + '%';
        el.style.top = (o.y * 100) + '%';
        el.style.borderColor = o.color || '#0af';
        container.appendChild(el);
      });
    }
  </script>
</body>
</html>

Step 5: Serve the static files from FastAPI

I need to mount the static folder so the browser can load the page. Add these lines to main.py near the top.

from fastapi.staticfiles import StaticFiles

app.mount("/static", StaticFiles(directory="static"), name="static")

Run it

Start the server with Uvicorn, then open http://localhost:8000/static/index.html on a phone or laptop with a camera.

uvicorn main:app --reload --host 0.0.0.0

I pointed my phone at a desk with a loose cable and typed: "I see a router with a blinking red light and a loose ethernet cable on the left side." The backend returned:

{
  "thought": "The blinking red light indicates no internet connection, likely caused by the unplugged cable.",
  "overlays": [
    { "label": "Plug in ethernet cable", "x": 0.25, "y": 0.55, "color": "#ff4444" },
    { "label": "Check power LED", "x": 0.60, "y": 0.40, "color": "#00ff88" }
  ]
}

The frontend rendered two labels directly over the video feed. Because Oxlo.ai uses request-based pricing, this interaction cost one flat request regardless of how verbose my scene description was.

Next steps

Wire in an Oxlo.ai vision model to auto-generate the scene description from camera frames before passing it to Llama 3.3 70B for overlay instructions. Or package the frontend as a PWA with the Web Share Target API so technicians can launch it directly from a work ticket.

Building Language Understanding Models with LLM: A Step-by-Step Guide

shashank ms — Mon, 27 Jul 2026 01:35:09 +0000

We are going to build a language understanding layer that turns raw customer support messages into structured intent and entity JSON. This kind of module is the backbone of any agentic support system, and it runs entirely on an LLM hosted through Oxlo.ai.

What you'll need

Before starting, make sure you have the following ready.

Python 3.10 or newer installed locally.
The OpenAI Python SDK. Install it with pip install openai.
An Oxlo.ai API key from https://portal.oxlo.ai. Oxlo.ai is a drop-in OpenAI-compatible provider, so the same SDK works without modification.

Step 1: Define the schema and system prompt

Language understanding starts with a strict contract. I write a system prompt that tells the model exactly what fields to return and what each field means. I also pin the output to JSON so downstream code can rely on the structure.

SYSTEM_PROMPT = """
You are a language understanding engine. Analyze the user's support message and return a single JSON object with these exact keys:

- intent: one of [refund_request, technical_issue, billing_question, account_access, general_inquiry]
- entities: an object containing any relevant identifiers such as order_id, email, or product_name
- urgency: one of [low, medium, high, critical]
- summary: a one-sentence summary of the user's core need
- next_action: one of [escalate_to_human, send_kb_link, process_refund, verify_identity]

Rules:
1. Do not include markdown formatting or explanation outside the JSON.
2. If an entity is missing, use null.
3. If the message is ambiguous, set intent to general_inquiry and urgency to medium.
"""

Step 2: Set up the Oxlo.ai client

Because Oxlo.ai exposes a fully OpenAI-compatible endpoint, I import the standard client and point it at Oxlo.ai. I will use the general-purpose Llama 3.3 70B model, though Oxlo.ai also offers Qwen 3 32B, Kimi K2.6, and DeepSeek V3.2 if you need multilingual or reasoning variants.

from openai import OpenAI

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key="YOUR_OXLO_API_KEY"  # Replace with your key from https://portal.oxlo.ai
)

Step 3: Build the extraction function

Now I wrap the API call in a small function. I set the temperature low to reduce creativity, and I enable JSON mode so the model knows it must emit valid JSON.

import json

def extract_intent(user_message: str) -> dict:
    response = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": user_message},
        ],
        temperature=0.1,
        response_format={"type": "json_object"},
    )
    raw = response.choices[0].message.content
    return json.loads(raw)

Step 4: Add schema validation

Even with JSON mode, I want to guard against missing keys. I use Pydantic to coerce and validate the model output so my application fails loudly and cleanly if the shape drifts.

from typing import Optional
from pydantic import BaseModel, Field

class SupportIntent(BaseModel):
    intent: str
    entities: dict
    urgency: str
    summary: str
    next_action: str

def parse_support_message(text: str) -> SupportIntent:
    raw_json = extract_intent(text)
    return SupportIntent.model_validate(raw_json)

Step 5: Batch process a conversation log

In production you rarely process one message at a time. Here is a short loop that classifies three incoming tickets and prints the structured results. This is where Oxlo.ai's request-based pricing shines: each message costs one request, regardless of how long the system prompt is.

tickets = [
    "I was charged twice for order #49281 and I need my money back immediately.",
    "Hey, how do I reset my password? I cannot log in.",
    "The app crashes every time I open the dashboard on my Pixel 7. This is blocking my work.",
]

for ticket in tickets:
    result = parse_support_message(ticket)
    print(result.model_dump_json(indent=2))

Run it

Save everything in a file named intent_parser.py, export your API key, and run python intent_parser.py. You should see structured output similar to this.

{
  "intent": "billing_question",
  "entities": {"order_id": "#49281"},
  "urgency": "high",
  "summary": "Customer was double-charged and wants a refund.",
  "next_action": "process_refund"
}
{
  "intent": "account_access",
  "entities": {},
  "urgency": "medium",
  "summary": "Customer needs help resetting their password.",
  "next_action": "send_kb_link"
}
{
  "intent": "technical_issue",
  "entities": {"device": "Pixel 7"},
  "urgency": "critical",
  "summary": "App crashes on dashboard open, blocking work.",
  "next_action": "escalate_to_human"
}

Next steps

Swap in qwen-3-32b or kimi-k2.6 if you need stronger multilingual or reasoning performance for the same pipeline. Because Oxlo.ai charges per request rather than per token, expanding the system prompt or adding few-shot examples does not increase your cost. See https://oxlo.ai/pricing for plan details.

From here, you can wire this parser into a FastAPI endpoint or connect it to a message queue like Redis Streams to build a real-time support agent. The module is just plain Python, so it drops into any async worker without vendor lock-in.

Deploying LLM Models on Cloud Functions: Best Practices and Strategies

shashank ms — Mon, 27 Jul 2026 01:33:20 +0000

Serverless platforms like AWS Lambda, Google Cloud Functions, and Azure Functions are attractive for LLM applications because they scale to zero and charge only for compute time used. However, deploying billion-parameter models directly inside a Cloud Function is usually impractical. Memory limits, execution timeouts, and cold-start latency make on-premise inference inside a function container a fragile pattern. The dominant best practice is to treat Cloud Functions as an orchestration and API normalization layer while offloading inference to a dedicated platform. Oxlo.ai fits this architecture precisely: it offers 45+ open-source and proprietary models with no cold starts on popular models, full OpenAI SDK compatibility, and request-based pricing that removes the cost uncertainty of token-based billing.

Separating Inference from Serverless Compute

Cloud Functions excel at short-lived, stateless tasks. A typical AWS Lambda deployment offers up to 10 GB of memory and a 15-minute timeout. While this is sufficient for application logic, it is inadequate for serving a 70B parameter model such as Llama 3.3 70B or DeepSeek R1 671B MoE. Loading these weights into a serverless container would trigger multi-minute cold starts on every scale-out event, and inference latency would be unpredictable.

The recommended architecture keeps models on dedicated GPU infrastructure and uses Cloud Functions for pre-processing, authentication, RAG retrieval, and response formatting. Oxlo.ai provides fully OpenAI API compatible endpoints at https://api.oxlo.ai/v1, so you can replace your OpenAI client configuration without rewriting application code. Because Oxlo.ai has no cold starts on popular models, the time-to-first-token remains consistent even when your serverless tier scales from zero.

Connection Pooling and Client Reuse

One of the most common mistakes in serverless LLM apps is initializing a new HTTP client inside every function invocation. This adds TLS handshake overhead and DNS latency on every request. Instead, initialize your client outside the handler so it persists across warm invocations.

import os
from openai import OpenAI

# Initialize once, reuse across invocations
client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key=os.environ.get("OXLO_API_KEY")
)

def lambda_handler(event, context):
    messages = event.get("messages", [])
    
    response = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=messages,
        stream=False
    )
    
    return {
        "statusCode": 200,
        "body": response.choices[0].message.content
    }

This pattern works identically in Google Cloud Functions,

Integrating LLMs with Web Applications: A Beginner's Guide

shashank ms — Sun, 26 Jul 2026 23:33:07 +0000

Adding a large language model to a web application can turn a static interface into an interactive assistant, but the path from API key to production feature is rarely obvious. Beginners often struggle with where to run inference, how to manage secrets, and how to stream output without freezing the UI. This guide walks through a practical, full-stack integration using standard HTTP and the OpenAI SDK pattern, with concrete code you can run today.

Architecture Overview

A typical LLM web integration has three layers. The client browser handles rendering and user input. A backend server, which you control, stores the API key and orchestrates calls. The inference provider runs the model. Never expose provider keys in frontend bundles. Instead, have your backend proxy requests so you can add retries, logging, and rate limiting.

Choosing an Inference Provider

You need a provider that supports the model you want, scales without cold starts, and fits your pricing model. Token-based billing can make long-context and agentic workloads unpredictable because costs grow with every word in the prompt. Oxlo.ai uses request-based pricing, which means one flat cost per API call regardless of input length. For applications that send large documents or multi-turn conversation history, this can dramatically simplify budgeting. Oxlo.ai offers 45+ models across chat, code, vision, and embeddings, is fully OpenAI SDK compatible, and requires no code changes beyond updating the base URL. You can start on the free tier with 60 requests per day and explore the full catalog before committing. See the latest plans at https://oxlo.ai/pricing.

Building the Backend

We will use Python with FastAPI. Install dependencies:

pip install fastapi uvicorn openai python-dotenv

Create a .env file:

OXLO_API_KEY=your_oxlo_key

main.py:

import os
from fastapi import FastAPI, HTTPException
from fastapi.middleware.cors import CORSMiddleware
from pydantic import BaseModel
from openai import OpenAI

app = FastAPI()

app.add_middleware(
    CORSMiddleware,
    allow_origins=["http://localhost:3000"],
    allow_credentials=True,
    allow_methods=["*"],
    allow_headers=["*"],
)

client = OpenAI(
    api_key=os.getenv("OXLO_API_KEY"),
    base_url="https://api.oxlo.ai/v1",
)

class ChatRequest(BaseModel):
    message: str

@app.post("/chat")
async def chat(req: ChatRequest):
    try:
        response = client.chat.completions.create(
            model="llama-3.3-70b",
            messages=[{"role": "user", "content": req.message}],
            stream=False,
        )
        return {"reply": response.choices[0].message.content}
    except Exception as e:
        raise HTTPException(status_code=500, detail=str(e))

Run with uvicorn main:app --reload. The backend now proxies requests to Oxlo.ai without exposing your key.

Wiring the Frontend

A minimal HTML page:

<!DOCTYPE html>
<html>
<head>
  <meta charset="UTF-8">
  <title>LLM Chat</title>
</head>
<body>
  <div id="chat"></div>
  <input id="msg" type="text" placeholder="Ask something..." />
  <button onclick="send()">Send</button>

  <script>
    async function send() {
      const input = document.getElementById('msg');
      const box = document.getElementById('chat');
      const res = await fetch('http://localhost:8000/chat', {
        method: 'POST',
        headers: {'Content-Type': 'application/json'},
        body: JSON.stringify({message: input.value})
      });
      const data = await res.json();
      const p = document.createElement('p');
      p.textContent = data.reply || data.error;
      box.appendChild(p);
      input.value = '';
    }
  </script>
</body>
</html>

Streaming Responses

Blocking UI updates feel slow. OpenAI SDK streaming works with Oxlo.ai by setting stream=True and returning an SSE stream.

Update the backend:

from fastapi.responses import StreamingResponse

@app.post("/chat/stream")
async def chat_stream(req: ChatRequest):
    def event_generator():
        stream = client.chat.completions.create(
            model="llama-3.3-70b",
            messages=[{"role": "user", "content": req.message}],
            stream=True,
        )
        for chunk in stream:
            text = chunk.choices[0].delta.content
            if text:
                yield f"data: {text}\n\n"
        yield "data: [DONE]\n\n"

    return StreamingResponse(event_generator(), media_type="text/event-stream")

Update the frontend to read the stream with fetch and a ReadableStream reader:

async function sendStream() {
  const input = document.getElementById('msg');
  const box = document.getElementById('chat');
  const p = document.createElement('p');
  box.appendChild(p);

  const res = await fetch('http://localhost:8000/chat/stream', {
    method: 'POST',
    headers: {'Content-Type': 'application/json'},
    body: JSON.stringify({message: input.value})
  });

  const reader = res.body.getReader();
  const decoder = new TextDecoder();
  let buffer = '';

  while (true) {
    const {done, value} = await reader.read();
    if (done) break;
    buffer += decoder.decode(value, {stream: true});
    const lines = buffer.split('\n');
    buffer = lines.pop();
    for (const line of lines) {
      if (line.startsWith('data: ')) {
        const payload = line.slice(6);
        if (payload === '[DONE]') return;
        p.textContent += payload;
      }
    }
  }
}

Adding Tool Use

Many applications need more than text. Function calling lets the model request actions. Oxlo.ai supports tool use on compatible models such as Qwen 3 32B and Llama 3.3 70B.

Define a tool schema:

tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get current weather for a city",
            "parameters": {
                "type": "object",
                "properties": {
                    "city": {"type": "string"}
                },
                "required": ["city"]
            }
        }
    }
]

response = client.chat.completions.create(
    model="qwen3-32b",
    messages=[{"role": "user", "content": req.message}],
    tools=tools,
    tool_choice="auto",
)

If the model returns a tool call, execute the function locally and send the result back in a second request. This pattern works identically on Oxlo.ai because the endpoints follow the OpenAI chat/completions schema.

Cost Control and Context Windows

When you build features that summarize PDFs, iterate over codebase files, or maintain long agentic memory, token-based bills scale with every extra sentence. Oxlo.ai’s request-based pricing removes that variable. A single request costs the same whether you send a one-line prompt or a full conversation transcript near the context limit. This makes long-context workflows, like analyzing 131K tokens with Kimi K2.6 or using DeepSeek V4 Flash with 1M context, far easier to forecast. You can compare plans at https://oxlo.ai/pricing.

For additional safety, always enforce a maximum request budget in your backend. Track daily usage in a simple counter or Redis key, and fallback to a smaller model if you approach your plan limit.

Conclusion

Integrating an LLM into a web application is fundamentally an API integration problem: secure the key on the backend, stream tokens to the frontend, and handle errors gracefully. By using an OpenAI SDK-compatible provider, you keep your stack portable and your code short. Oxlo.ai fits this workflow with a drop-in base URL change, request-based pricing that favors long-context applications, and a free tier that lets you experiment before deploying. Start with the examples above, point your client to https://api.oxlo.ai/v1, and iterate.

Optimizing LLM Inference for Code Analysis

shashank ms — Sun, 26 Jul 2026 23:32:14 +0000

Code analysis pipelines, whether for static review, bug detection, or automated refactoring, push LLM inference into regimes that differ from typical chat workloads. Context windows fill with entire file trees, dependency graphs, and multi-turn diffs. Token counts balloon quickly, and latency becomes the bottleneck between a developer and actionable feedback. Optimizing these pipelines requires more than choosing a large model. It demands a strategy around context management, model selection, and pricing mechanics that align cost with the actual shape of code workloads.

Why Code Analysis Is Different

Code analysis tasks differ from conversational AI because inputs are rarely short. A single request might include a full source file, surrounding context from an import chain, and instructions to detect security vulnerabilities or suggest refactors. Unlike Q&A bots that average a few hundred tokens per turn, code agents routinely submit thousands or tens of thousands of tokens in a single pass. This asymmetry means that input token volume, not output generation, usually dominates latency and cost.

The Cost of Long Context

Token-based billing penalizes exactly the behavior that makes code analysis effective: providing broad context. When a provider charges per input and output token, analyzing a large legacy file or a multi-file diff becomes expensive fast. The cost scales linearly with the size of the codebase under inspection, which discourages thoroughness.

Oxlo.ai uses request-based pricing: one flat cost per API call regardless of prompt length. For code analysis, where long context is not a luxury but a requirement, this model removes the penalty on input size. You can pass entire files, surrounding modules, and system prompts without watching token meters run. For teams running static analysis at scale, the difference can be an order of magnitude or more compared to token-based inference. See Oxlo.ai pricing for plan details.

Model Selection

Not every code task requires the largest model. The right choice depends on whether you are parsing syntax, reasoning about architecture, or generating patches.

Fast linting and syntax checks: Oxlo.ai Coder Fast and Qwen 3 Coder 30B handle structured code tasks with low latency.
General review and refactoring: Llama 3.3 70B and DeepSeek V3.2 provide strong reasoning across languages.
Complex debugging and deep reasoning: DeepSeek R1 671B MoE and Kimi K2.6 excel at chain-of-thought analysis for subtle bugs.
Agentic workflows: GLM 5 and Minimax M2.5 support long-horizon tool use when analysis pipelines need to invoke tests, linters, or search tools.

Oxlo.ai hosts 45+ models across these categories with no cold starts, so you can route requests to the smallest sufficient model without provisioning overhead.

Prompt Engineering for Code

Context windows on modern models are large, but filling them indiscriminately wastes inference time and can dilute attention. A focused prompt beats a bloated one.

Use hierarchical prompts. Start with a high-level goal, then attach the relevant file, then add cross-file context only when needed.
Strip noise. Remove comments, auto-generated headers, and irrelevant methods before submission. Fewer tokens mean lower latency, even on request-based platforms.
Specify output format in the system prompt. If you need JSON, declare the schema upfront. This reduces parsing errors and follow-up requests.

Chunking and Retrieval

When a repository exceeds the context window, or when you want to minimize per-request latency, chunk the codebase intelligently. Rather than sending an entire monorepo, index files by module and retrieve only the snippets relevant to the query.

A simple retrieval-augmented generation pipeline for code might look like this:

from openai import OpenAI
import os

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key=os.environ["OXLO_API_KEY"]
)

def analyze_chunk(file_path: str, chunk: str, instruction: str):
    response = client.chat.completions.create(
        model="oxlo.ai-coder-fast",
        messages=[
            {"role": "system", "content": "You are a code reviewer. Respond with JSON."},
            {"role": "user", "content": f"File: {file_path}\n\n{chunk}\n\nInstruction: {instruction}"}
        ],
        response_format={"type": "json_object"}
    )
    return response.choices[0].message.content

# Analyze each chunk independently, then aggregate

Because Oxlo.ai charges per request, splitting a large file into multiple chunks and processing them in parallel does not incur per-token penalties. You pay per call, which makes batching and map-reduce patterns predictable.

Structured Output and Tool Use

Code analysis pipelines rarely consume raw prose. They need line numbers, severity scores, and suggested diffs. Oxlo.ai supports JSON mode and function calling across its chat models, so you can constrain outputs to machine-readable schemas.

For example, you can define a tool that files a ticket or triggers a linter, and let the model decide when to invoke it:

tools = [
    {
        "type": "function",
        "function": {
            "name": "run_linter",
            "description": "Run the project linter on a file",
            "parameters": {
                "type": "object",
                "properties": {
                    "file_path": {"type": "string"}
                },
                "required": ["file_path"]
            }
        }
    }
]

response = client.chat.completions.create(
    model="qwen3-32b",
    messages=[{"role": "user", "content": "Review src/auth.js for style violations."}],
    tools=tools,
    tool_choice="auto"
)

Structured output eliminates regex parsing and reduces the number of round trips required to integrate LLM results into CI/CD pipelines.

Caching and State Management

Multi-turn code reviews, where the model refines suggestions based on developer feedback, can accumulate context quickly. Preserve conversation state client-side and append only new diffs or comments rather than resending the entire file history.

When using Oxlo.ai, because cost is request-based, you only pay for the new API call, not for the cumulative tokens in the conversation history. This encourages longer, more