DEV Community: shashank ms

Building a Secure LLM System: Essential Considerations

shashank ms — Sun, 16 Aug 2026 15:34:49 +0000

Building a production-grade LLM system requires more than selecting a capable model. Security must be embedded at every layer, from the initial user prompt through the inference provider to the final rendered output. A compromised pipeline can expose sensitive data, allow prompt injection attacks, or create unpredictable cost spikes that destabilize your infrastructure. This article covers the essential architectural considerations for securing LLM integrations, with practical patterns you can implement today.

Input Sanitization and Prompt Injection Defense

Prompt injection remains one of the most common attack vectors against LLM applications. Attackers embed malicious instructions inside user-controlled input to override system prompts or extract training data. A robust defense starts with strict input validation. Treat all user content as untrusted, and isolate system instructions from user variables.

Use allowlists for expected input patterns, and enforce length limits before any text reaches the model. When possible, structure prompts so that user content is clearly delimited. For example, wrap user input in XML or JSON tags that the model can distinguish from system logic.

# Unsafe: direct concatenation
system_prompt = "You are a helpful assistant. Answer the question: " + user_input

# Safer: explicit role separation with delimiters
messages = [
    {"role": "system", "content": "You are a helpful assistant. Only answer questions about public documentation."},
    {"role": "user", "content": f"<user_query>{sanitize(user_input)}</user_query>"}
]

Even with sanitization, never execute model outputs directly in privileged environments. Any code, SQL, or shell commands generated by an LLM should pass through a secondary validation layer or human review before execution.

Output Validation and Structured Generation

Raw text generation is difficult to validate programmatically. Structured output formats reduce the attack surface by constraining what the model can return. JSON mode and function calling let you define schemas that outputs must adhere to, making downstream parsing safer and more predictable.

Oxlo.ai supports JSON mode and function calling across its chat and reasoning models, including Llama 3.3 70B, Qwen 3 32B, and DeepSeek R1 671B MoE. By enforcing a schema, you prevent the model from emitting unexpected markup or instructions that could confuse your client application.

import openai

client = openai.OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key=os.environ["OXLO_API_KEY"]
)

response = client.chat.completions.create(
    model="llama-3.3-70b",
    messages=[{"role": "user", "content": "Extract the name and email"}],
    response_format={"type": "json_object"},
    tools=[{
        "type": "function",
        "function": {
            "name": "extract_contact",
            "parameters": {
                "type": "object",
                "properties": {
                    "name": {"type": "string"},
                    "email": {"type": "string"}
                },
                "required": ["name", "email"]
            }
        }
    }],
    tool_choice="auto"
)

Always validate the parsed JSON against your schema on the client side. A model may hallucinate keys or types, so treat the output as untrusted data until verified.

API Security and Key Management

Your API credentials are the keys to your inference infrastructure. Store them in environment variables or a secrets manager, never in source control. Rotate keys quarterly, and use separate keys for development, staging, and production environments.

Because Oxlo.ai is fully OpenAI SDK compatible, you can switch to Oxlo.ai by changing only the base URL and API key. This drop-in compatibility means your existing secret management patterns, retry logic, and error handling require no refactoring.

import openai

client = openai.OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key=os.environ["OXLO_API_KEY"]  # Loaded from vault or env var
)

Enable request signing or additional proxy authentication if your architecture demands it. If you expose LLM features to end users, proxy requests through your backend rather than embedding API keys in client-side code.

Model Supply Chain and Inference Integrity

Using open-source models reduces vendor lock-in, but it introduces supply chain risks. You must trust that the weights running on your provider's infrastructure match the published hashes and have not been tampered with. Choose inference platforms that load official model artifacts and offer consistent, reproducible behavior.

Oxlo.ai hosts 45+ open-source and proprietary models across seven categories, including code, vision, audio, and embeddings. Popular models such as DeepSeek V4 Flash, Kimi K2.6, and GLM 5 run with no cold starts, so you receive deterministic response times without the jitter caused by lazy container initialization. Predictable infrastructure behavior makes anomaly detection and security monitoring easier.

When evaluating a provider, verify that they expose standard model identifiers and do not silently swap versions. Oxlo.ai uses explicit versioning in model names, so a request to deepseek-r1-671b today returns the same weights tomorrow.

Cost Predictability and Operational Security

Security is not only about preventing breaches. It is also about preventing operational surprises. Token-based pricing creates a direct financial incentive for adversaries to craft long inputs that inflate your bill. A malicious user who discovers an unprotected endpoint can stream thousands of tokens per request, turning a small integration into a major cost center.

Oxlo.ai uses request-based pricing with one flat cost per API request regardless of prompt length. Unlike token-based providers such as Together AI, Fireworks AI, OpenRouter, Replicate, and Anyscale, your cost does not scale with input length. This makes Oxlo.ai significantly cheaper for long-context and agentic workloads, and it eliminates the risk of token-length denial-of-wallet attacks. An adversary can still spam requests, but each hit is capped at a known unit cost, which simplifies rate limiting and budget enforcement. For detailed plan information, see the Oxlo.ai pricing page.

# With request-based pricing, long system prompts and multi-turn history
# do not trigger unexpected cost spikes.

messages = [
    {"role": "system", "content": open("large_context.txt").read()},  # 20K tokens
    {"role": "user", "content": "Summarize the key points."}
]

# Cost remains predictable regardless of input size.
response = client.chat.completions.create(
    model="qwen-3-32b",
    messages=messages
)

Auditing and Logging

Comprehensive logging is essential for incident response. Record request metadata, model identifiers, and output fingerprints. Avoid logging full prompt content if it contains personally identifiable information, but retain enough context to detect abuse patterns.

Stream responses through your application layer so you can intercept and audit chunks in real time. Oxlo.ai supports streaming responses across its endpoints, allowing you to build middleware that flags sensitive content or anomalous patterns before they reach the user.

Implement rate limiting per user and per IP. Use exponential backoff for retries, and alert on error rate spikes that might indicate an attack against your endpoint.

Conclusion

Securing an LLM system demands defense in depth. Sanitize inputs, constrain outputs, protect your API keys, audit your model supply chain, and remove financial attack vectors through predictable pricing. Oxlo.ai provides a developer-first inference platform with OpenAI SDK compatibility, no cold starts, and request-based pricing that caps your exposure to input-length abuse. Whether you are running agentic workflows with GLM 5, coding assistants with Qwen 3 Coder 30B, or vision pipelines with Kimi VL A3B, Oxlo.ai fits naturally into a security-conscious architecture.

Building Robust CI/CD Pipelines for LLM Applications

shashank ms — Sun, 16 Aug 2026 11:35:23 +0000

I recently shipped an automated code review agent that runs inside our CI pipeline. It reads git diffs and flags potential bugs, missing tests, and style issues before a human reviewer opens the pull request. In this tutorial, I will walk through the exact Python script and GitHub Actions workflow I use, powered by Oxlo.ai inference.

What you'll need

Python 3.10 or newer
An Oxlo.ai API key from https://portal.oxlo.ai
A Git repository with at least one commit
The OpenAI SDK: pip install openai
A GitHub repository if you want to run the final CI step

Step 1: Scaffold the review script

We start with a single Python file that loads the Oxlo.ai client and accepts a diff via stdin. This keeps the agent stateless and easy to invoke from any CI runner.

import os
import sys
from openai import OpenAI

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key=os.environ["OXLO_API_KEY"]
)

def get_diff():
    return sys.stdin.read()

if __name__ == "__main__":
    diff = get_diff()
    if not diff.strip():
        print("No diff provided.")
        sys.exit(0)

Step 2: Write the system prompt

The system prompt is the only configuration the agent needs. I keep it in a separate variable so I can tune it without touching the logic.

SYSTEM_PROMPT = """You are a senior staff engineer performing code review.
Review the provided git diff and output a JSON object with exactly two keys:
- "issues": a list of objects, each with "severity" (critical, warning, or note), "file", "line", and "message".
- "summary": a one-sentence overview of the change.

Be concise. Only flag real problems: logic errors, missing error handling, security risks, or unclear naming. Do not comment on formatting unless it hurts readability."""

Step 3: Call Oxlo.ai with JSON mode

I use Llama 3.3 70B because it follows structured instructions reliably and runs without cold starts on Oxlo.ai. We enable JSON mode and parse the response so the CI runner can act on it.

import json

def review_diff(diff_text: str):
    response = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": f"Review this diff:\n\n{diff_text}"},
        ],
        response_format={"type": "json_object"},
        temperature=0.2,
    )
    raw = response.choices[0].message.content
    return json.loads(raw)

if __name__ == "__main__":
    diff = get_diff()
    result = review_diff(diff)
    print(json.dumps(result, indent=2))

Step 4: Add CI-friendly exit codes

A pipeline step needs to pass or fail. I count critical issues and return a non-zero exit code when any are found, which blocks the merge until a human overrides.

def report_and_exit(result: dict):
    issues = result.get("issues", [])
    critical_count = sum(1 for i in issues if i.get("severity") == "critical")

    for issue in issues:
        icon = {"critical": "❌", "warning": "⚠️", "note": "ℹ️"}.get(issue["severity"], "•")
        print(f"{icon} [{issue['severity'].upper()}] {issue['file']}:{issue.get('line', '?')} - {issue['message']}")

    print(f"\nSummary: {result.get('summary', 'No summary provided.')}")
    print(f"Found {critical_count} critical issue(s).")

    if critical_count > 0:
        sys.exit(1)
    sys.exit(0)

if __name__ == "__main__":
    diff = get_diff()
    result = review_diff(diff)
    report_and_exit(result)

Step 5: Containerize for reproducible CI runs

CI runners should not depend on the host Python environment. A minimal Dockerfile lets us pin the OpenAI SDK version and run the same image locally and in the cloud.

FROM python:3.11-slim

WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY review.py .
ENTRYPOINT ["python", "review.py"]

Save this requirements file in the same folder.

openai>=1.0

Build and test locally before pushing.

docker build -t llm-review-agent .
git diff HEAD~1 | docker run --rm -e OXLO_API_KEY=$OXLO_API_KEY -i llm-review-agent

Step 6: Wire into a GitHub Actions workflow

The final piece is a workflow that triggers on pull requests, feeds the diff to the Oxlo.ai-powered agent, and posts the results inline. Because Oxlo.ai uses flat per-request pricing, the cost of reviewing large diffs is predictable, which matters when this runs on every push.

name: LLM Code Review

on:
  pull_request:
    types: [opened, synchronize]

jobs:
  review:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - name: Build review agent
        run: docker build -t llm-review-agent .

      - name: Run Oxlo.ai review on PR diff
        env:
          OXLO_API_KEY: ${{ secrets.OXLO_API_KEY }}
        run: |
          git diff origin/${{ github.base_ref }}...HEAD | \
            docker run --rm -e OXLO_API_KEY -i llm-review-agent

Run it

Here is the complete review.py assembled from the steps above. Export your Oxlo.ai key and pipe any git diff into it.

import os
import sys
import json
from openai import OpenAI

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key=os.environ["OXLO_API_KEY"]
)

SYSTEM_PROMPT = """You are a senior staff engineer performing code review.
Review the provided git diff and output a JSON object with exactly two keys:
- "issues": a list of objects, each with "severity" (critical, warning, or note), "file", "line", and "message".
- "summary": a one-sentence overview of the change.

Be concise. Only flag real problems: logic errors, missing error handling, security risks, or unclear naming. Do not comment on formatting unless it hurts readability."""

def get_diff():
    return sys.stdin.read()

def review_diff(diff_text: str):
    response = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": f"Review this diff:\n\n{diff_text}"},
        ],
        response_format={"type": "json_object"},
        temperature=0.2,
    )
    raw = response.choices[0].message.content
    return json.loads(raw)

def report_and_exit(result: dict):
    issues = result.get("issues", [])
    critical_count = sum(1 for i in issues if i.get("severity") == "critical")

    for issue in issues:
        icon = {"critical": "❌", "warning": "⚠️", "note": "ℹ️"}.get(issue["severity"], "•")
        print(f"{icon} [{issue['severity'].upper()}] {issue['file']}:{issue.get('line', '?')} - {issue['message']}")

    print(f"\nSummary: {result.get('summary', 'No summary provided.')}")
    print(f"Found {critical_count} critical issue(s).")

    if critical_count > 0:
        sys.exit(1)
    sys.exit(0)

if __name__ == "__main__":
    diff = get_diff()
    if not diff.strip():
        print("No diff provided.")
        sys.exit(0)
    result = review_diff(diff)
    report_and_exit(result)

Test it against the last commit.

export OXLO_API_KEY="sk-oxlo.ai-..."
git diff HEAD~1 | python review.py

Example output from a real review.

⚠️ [WARNING] auth.py:42 - Hardcoded timeout may cause flaky tests under high load.
ℹ️ [NOTE] auth.py:55 - Consider renaming `do_thing` to `validate_token`.

Summary: Adds bearer token validation to the auth middleware but introduces a hardcoded timeout.
Found 0 critical issue(s).

Wrap-up

You can extend this agent by splitting large diffs into file chunks and calling Oxlo.ai in parallel. The flat per-request pricing means fanning out across ten files costs the same as one, which keeps the pipeline economical. Another solid next step is to cache results in Redis keyed by commit SHA so reruns of identical diffs do not burn requests.

Using LLM for Model Monitoring and Maintenance: Best Practices

shashank ms — Sun, 16 Aug 2026 11:34:39 +0000

Production LLM systems fail silently. A prompt change, a shift in user behavior, or a subtle regression in upstream data can degrade output quality without triggering traditional alarms. Conventional ML monitoring, built around scalar metrics and fixed schemas, struggles with the open-ended nature of generative models. Using an LLM to monitor your models, a pattern often called LLM-as-a-judge, gives you semantic understanding of drift and failure that rule-based checks cannot capture. The challenge is doing it at a cost and latency that does not exceed the system being monitored.

The Case for LLM-Powered Monitoring

Classical monitoring looks for distribution shift in embeddings or input features. That is necessary but not sufficient. When a model generates free text, code, or images, the failure modes are semantic: hallucinations, policy violations, incorrect tool use, or style regressions. An LLM judge can parse these dimensions because it understands context and intent.

Key tasks where an LLM monitor outperforms static rules include:

Output quality scoring for correctness, helpfulness, and tone
Adversarial and jailbreak detection
Root-cause analysis of failure clusters
Automated labeling for fine-tuning datasets

Architecture Patterns

A production monitoring pipeline usually combines several layers. Each layer trades off latency, cost, and depth of analysis.

LLM-as-Judge. A stronger reasoning model evaluates outputs from your production model. For nuanced reasoning, models like DeepSeek R1 671B MoE or Kimi K2.6 on Oxlo.ai provide deep chain-of-thought evaluation. You send the production trace, and the judge returns a structured score.

Embedding Drift. Compute embeddings over windows of production traffic using BGE-Large or E5-Large. Detect cosine-distance drift between baseline and current distributions. This is cheap and stateless.

Agentic Review. For

LLMops vs MLOps: Making the Right Choice for Your AI Workflow

shashank ms — Sun, 16 Aug 2026 11:33:02 +0000

AI infrastructure has split into two distinct operational cultures. MLOps, born from classical machine learning, focuses on feature stores, training pipelines, and model versioning for predictive workloads. LLMOps, the newer discipline, handles prompt engineering, context management, and inference orchestration for large language models. Understanding where each paradigm applies saves engineering teams from forcing transformer-based workflows into traditional ML pipelines, or vice versa.

What Is MLOps?

MLOps is the practice of operationalizing classical machine learning models at scale. It covers data validation, feature engineering, experiment tracking, model training, and deployment of predictive services. Teams using MLOps typically manage smaller model artifacts, version datasets, and monitor statistical drift in production. Common use cases include fraud detection, recommendation engines, and time-series forecasting.

What Is LLMOps?

LLMOps is the discipline of deploying and managing large language models in production. Instead of retraining weights for every change, practitioners version prompts, manage retrieval-augmented generation pipelines, and orchestrate agent tool use. The focus shifts from gradient descent to context engineering, with production concerns centered on latency, throughput, and inference cost across long-context or multi-turn sessions.

Key Architectural Differences

Primary artifact. MLOps versions model weights and feature pipelines. LLMOps versions prompts, system instructions, and tool schemas.
Compute profile. MLOps often requires GPU clusters for training. LLMOps is dominated by inference serving, where context length drives cost and latency.
Observability. MLOps tracks feature drift and prediction accuracy. LLMOps tracks input/output quality, hallucination rates, and tool-call reliability.
Cost structure. MLOps capex is front-loaded on training infrastructure. LLMOps opex scales with every API call, making pricing models a first-class design decision.

When to Choose MLOps vs LLMOps

If your workload involves structured tabular data, deterministic regression, or computer vision models under fifty million parameters, MLOps remains the correct foundation. If your application depends on multi-step reasoning, document analysis, or agentic tool chains, LLMOps is unavoidable. Many enterprises run both in parallel. The mistake is forcing an LLM into a classical model registry or, conversely, running a gradient-boosted classifier through a chat-completions endpoint.

The Inference Layer in LLMOps

A production LLMOps stack needs broad model access, reliable tool use, and predictable billing. Oxlo.ai is a developer-first AI inference platform with request-based pricing: one flat cost per API request regardless of prompt length. Unlike token-based providers such as Together AI, Fireworks AI, OpenRouter, Replicate, and Anyscale, cost does not scale with input length, so Oxlo.ai is significantly cheaper for long-context and agentic workloads.

The platform hosts 45+ open-source and proprietary models across 7 categories, including chat and reasoning models like Qwen 3, Llama 3/4, DeepSeek R1 and V3, Kimi K2.x, GPT-Oss, Mistral, GLM 5, and Minimax. It also offers code models, vision models, image generation, audio, embeddings, and object detection. All endpoints are fully OpenAI SDK compatible, with no cold starts, streaming responses, function calling, JSON mode, and vision support. For teams building LLMOps pipelines, this means you can swap in Oxlo.ai without rewriting client code.

Unified API for Both Paradigms

Because Oxlo.ai exposes an OpenAI-compatible endpoint, you can call it from existing MLOps or LLMOps tooling with minimal changes. The following example uses the Python SDK to stream a reasoning request:

import openai

client = openai.OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key="YOUR_OXLO_API_KEY"
)

response = client.chat.completions.create(
    model="llama-3.3-70b",
    messages=[
        {"role": "system", "content": "You are a precise coding assistant."},
        {"role": "user", "content": "Refactor this function to use async/await."}
    ],
    stream=True
)

for chunk in response:
    print(chunk.choices[0].delta.content or "", end="")

Switching to a reasoning specialist like DeepSeek R1 671B MoE or a long-context model like Kimi K2.6 requires only changing the model string. This flexibility is critical in LLMOps, where prompt chains may route to different model families based on task complexity.

Bridging MLOps and LLMOps

Hybrid architectures are increasingly common. A classical MLOps pipeline

LLM and DevOps: Integrating Large Language Models into DevOps Workflows

shashank ms — Sun, 16 Aug 2026 09:34:29 +0000

We are going to build a deployment triage agent that reads raw CI/CD or Kubernetes logs and returns a structured JSON diagnosis. This saves time during incidents by letting an LLM do the initial log reading and severity classification before a human opens the dashboard. The whole thing runs against Oxlo.ai using the standard OpenAI SDK, so it drops into existing Python tooling without new dependencies.

What you'll need

Python 3.10 or newer
The OpenAI Python SDK: pip install openai
An Oxlo.ai API key from https://portal.oxlo.ai

Sign up for the Free plan if you want to test without a credit card. Oxlo.ai uses flat per-request pricing, which keeps costs predictable even when you feed long stack traces into the model. See https://oxlo.ai/pricing for details.

Step 1: Bootstrap the Oxlo.ai client

First, I verify that I can reach the Oxlo.ai API and that my key is working. I use the standard OpenAI SDK and point it at the Oxlo.ai base URL.

from openai import OpenAI
import os

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key=os.environ.get("OXLO_API_KEY")
)

response = client.chat.completions.create(
    model="llama-3.3-70b",
    messages=[{"role": "user", "content": "Say OK"}],
)

print(response.choices[0].message.content)

Step 2: Write the DevOps system prompt

The system prompt is the only manual tuning I do. It tells the model how to format output and what to look for in infrastructure logs.

SYSTEM_PROMPT = """You are a DevOps triage assistant. Your job is to analyze deployment or application logs and produce a structured diagnosis.

Follow these rules:
1. Classify severity as one of: critical, warning, info.
2. Identify the root cause in one sentence.
3. List up to three concrete remediation steps.
4. Output strictly valid JSON with keys: severity, root_cause, remediation_steps (array), needs_human_escalation (boolean).

Be concise. Do not include markdown formatting inside the JSON."""

Step 3: Ingest and format logs

I write a small helper that reads a log file and wraps it in a clear instruction so the model knows what to analyze. I truncate extremely large files to the last 100 lines to stay well within context limits.

def load_log(path: str, tail: int = 100) -> str:
    with open(path, "r", encoding="utf-8", errors="ignore") as f:
        lines = f.readlines()
    return "".join(lines[-tail:])


def build_user_message(log_path: str) -> str:
    raw = load_log(log_path)
    return f"Analyze the following deployment log and return JSON only.\n\n

```\n{raw}\n```

"

Step 4: Run structured triage

Now I wire the pieces together. I call the Oxlo.ai chat endpoint with JSON mode enabled so the response is guaranteed to be parseable. I use llama-3.3-70b here because it handles instruction following and structured output reliably, but you can swap in qwen-3-32b or kimi-k2.6 if you need deeper reasoning.

import json

def triage(log_path: str) -> dict:
    user_message = build_user_message(log_path)

    response = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": user_message},
        ],
        response_format={"type": "json_object"},
        temperature=0.2,
    )

    content = response.choices[0].message.content
    return json.loads(content)


if __name__ == "__main__":
    result = triage("deploy.log")
    print(json.dumps(result, indent=2))

Step 5: Add a CI/CD gate

To make this useful in a pipeline, I turn the script into a small CLI that exits with a non-zero code on critical findings. This lets me block or flag deployments automatically.

import sys
import argparse

def main():
    parser = argparse.ArgumentParser(description="Triage deployment logs via Oxlo.ai")
    parser.add_argument("--log", required=True, help="Path to the log file")
    parser.add_argument("--output", default="triage.json", help="Where to write the JSON report")
    args = parser.parse_args()

    result = triage(args.log)

    with open(args.output, "w", encoding="utf-8") as f:
        json.dump(result, f, indent=2)

    print(f"Triage written to {args.output}")

    if result.get("severity") == "critical" and result.get("needs_human_escalation"):
        print("Critical issue detected. Failing the pipeline.")
        sys.exit(1)

    sys.exit(0)


if __name__ == "__main__":
    main()

Run it

Save a sample Kubernetes crash-loop log as deploy.log and run the agent.

export OXLO_API_KEY="sk-oxlo.ai-..."
python triage.py --log deploy.log --output report.json

Example output written to report.json:

{
  "severity": "critical",
  "root_cause": "The application container is crashing due to a missing DATABASE_URL environment variable.",
  "remediation_steps": [
    "Add the DATABASE_URL secret to the deployment manifest.",
    "Verify the ConfigMap is mounted in the correct namespace.",
    "Restart the deployment and check pod readiness."
  ],
  "needs_human_escalation": false
}

Because Oxlo.ai charges a flat rate per request, running this against a 500-line stack trace costs the same as a one-line summary. That makes it practical to run on every failed build without worrying about token math.

Next steps

Two concrete ways to extend this. First, add function calling so the agent can open a GitHub issue or trigger a rollback via your existing API. Second, switch to deepseek-v3.2 or kimi-k2.6 if you start feeding the agent multi-file build artifacts and need stronger reasoning across long contexts. Both are available on Oxlo.ai with the same request-based pricing and OpenAI-compatible SDK.

Using LLM on Cloud Platforms: Best Practices and Considerations

shashank ms — Sun, 16 Aug 2026 09:33:13 +0000

Deploying large language models in production means choosing a cloud backend that matches your latency, cost, and model diversity requirements. The market spans hyperscale providers and specialized inference hosts, including token-based services such as Together AI, Fireworks AI, OpenRouter, Replicate, and Anyscale. Each platform varies in pricing mechanics, cold-start behavior, and API compatibility. This article outlines practical best practices for running LLMs on cloud platforms and explains where a request-based alternative such as Oxlo.ai can simplify operations.

Evaluate Cost Predictability

For applications with long system prompts, retrieval-augmented generation, or agentic loops, token-based billing can create unpredictable spend. Every input token incurs cost, so a 100K context window or a multi-step agent workflow multiplies expenses quickly. A request-based model, by contrast, charges one flat fee per API call regardless of prompt length. Oxlo.ai uses this approach, which makes it significantly cheaper for long-context and agentic workloads because cost does not scale with input length. If your traffic is characterized by variable prompt sizes, evaluate whether a flat per-request structure reduces variance in your monthly bill. You can compare plans at https://oxlo.ai/pricing.

Model Availability and Portability

Production pipelines rarely rely on a single model. You may need a general-purpose chat model for customer support, a vision model for document parsing, an embedding model for search, and a code model for internal tooling. A cloud platform should expose these through a unified endpoint to avoid fragmented integrations. Oxlo.ai hosts more than 45 open-source and proprietary models across seven categories, including chat and reasoning, code, vision, image generation, audio, embeddings, and object detection. Flagship options include DeepSeek R1 671B MoE for deep reasoning, Kimi K2.6 for agentic coding and vision, and Qwen 3 32B for multilingual agent workflows. Because the service is fully OpenAI SDK compatible, switching from another provider is a one-line configuration change.

import os
import openai

client = openai.OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key=os.environ["OXLO_API_KEY"]
)

response = client.chat.completions.create(
    model="llama-3.3-70b",
    messages=[{"role": "user", "content": "Explain request-based pricing."}],
    stream=True
)

for chunk in response:
    print(chunk.choices[0].delta.content or "", end="")

Latency and Cold Start Tradeoffs

Serverless inference platforms often scale to zero to save GPU hours, but that optimization introduces cold-start latency. In user-facing chat or real-time agent loops, even a few seconds of initialization time degrades experience. When selecting a cloud provider, verify whether popular models are kept warm or if you must provision reserved capacity to avoid delays. Oxlo.ai maintains no cold starts on popular models, which means first-token latency remains consistent whether you are sending one request or one thousand.

API Compatibility and Migration Path

Adopting a new inference backend should not require rewriting client libraries or abandoning existing prompt templates. OpenAI SDK compatibility has become the de facto standard, allowing teams to migrate by changing only the base URL and API key. Oxlo.ai implements the full chat/completions, embeddings, images/generations, audio/transcriptions, and audio/speech endpoints, so existing Python, Node.js, or cURL scripts work without modification. This drop-in replacement pattern reduces vendor lock-in and accelerates experimentation.

Security and Data Governance

Cloud AI platforms should expose HTTPS-only endpoints, support API key rotation, and offer private deployment options for regulated workloads. For teams that need physical isolation, Oxlo.ai provides an Enterprise tier with dedicated GPUs and custom contracts. Even on shared infrastructure, traffic is encrypted in transit, and you retain control over whether to enable features such as streaming or JSON mode on a per-request basis.

Workload-Specific Tuning

Modern LLM applications depend on more than plain text generation. Function calling, structured JSON output, vision inputs, and multi-turn conversation state are baseline requirements. Before committing to a provider, confirm that these features are supported across the models you intend to use. Oxlo.ai supports streaming responses, function calling and tool use, JSON mode, vision inputs, and multi-turn conversations. When you build agents that chain tool calls or parse structured output, these capabilities eliminate the need for secondary parsing layers.

Conclusion

Running LLMs on cloud platforms requires balancing cost structure, model breadth, latency, and API ergonomics. Token-based providers remain viable for short-prompt workloads, but teams running long-context applications, agentic workflows, or multimodal pipelines should weigh the benefits of flat per-request pricing. Oxlo.ai offers a developer-first stack with more than 45 models, no cold starts, and full OpenAI SDK compatibility, making it a relevant option for teams that want predictable costs and minimal integration friction. Review the latest plans and model catalog at https://oxlo.ai/pricing to see how it fits your architecture.

Deploying LLM Models on Edge Devices: A Comprehensive Guide

shashank ms — Sun, 16 Aug 2026 09:32:17 +0000

Deploying large language models on edge devices is no longer theoretical. Engineers are shipping quantized Llama and Qwen variants to factory floors, mobile handsets, and on-premise servers to cut latency, preserve privacy, and keep critical workloads running offline. Yet edge hardware imposes hard limits on memory, thermals, and model size. The practical approach is rarely edge-only or cloud-only. It is a hybrid architecture where lightweight models run locally and demanding tasks route to a high-performance inference backend.

Understanding Edge Constraints

Edge devices range from Raspberry Pi 5 units with 8 GB RAM to NVIDIA Jetson AGX Orin boards with 64 GB. A full-precision Llama 3 8B requires roughly 16 GB of VRAM, which immediately rules out most embedded systems. Quantization is non-negotiable. INT4 and INT8 formats, including GGUF and ONNX variants, can squeeze a 7B parameter model into 4-6 GB. Even then, inference speed on CPU-only nodes often sits below 5 tokens per second. You must decide what runs locally and what does not.

Model Selection and Quantization

For edge deployment, parameter count matters more than benchmark hype. Models like Qwen 2.5 7B, Llama 3.1 8B, and Gemma 3 4B are common starting points. Use llama.cpp for cross-platform CPU and GPU inference, or ONNX Runtime for optimized execution on ARM and x86 edge nodes. On Apple silicon, MLX provides efficient inference with unified memory.

Example: Running a quantized model locally with llama.cpp via Python.

from llama_cpp import Llama

# Load a 4-bit quantized Qwen 2.5 7B Instruct
llm = Llama(
    model_path="./qwen2.5-7b-instruct-q4_k_m.gguf",
    n_ctx=4096,
    n_threads=4,
    verbose=False
)

output = llm.create_chat_completion(
    messages=[{"role": "user", "content": "Summarize this sensor log."}]
)
print(output["choices"][0]["message"]["content"])

If your edge hardware lacks the headroom for even a 7B model, consider sub-3B options such as Phi-3 Mini or Qwen 2.5 3B. They sacrifice reasoning depth but remain responsive for classification and extraction tasks.

Deployment Frameworks and Tooling

Several frameworks dominate edge LLM deployment:

llama.cpp: The universal workhorse. Supports GGUF, CPU/GPU hybrid offloading, and runs on Linux, macOS, Windows, and iOS.
Ollama: Wraps llama.cpp in a simple CLI and REST API. Ideal for rapid prototyping on edge gateways.
ONNX Runtime: Best when you need to target specific NPUs or DSPs, such as Qualcomm Hexagon or Intel Movidius.
TensorRT-LLM / TensorRT: Required for maximizing throughput on NVIDIA Jetson and discrete edge GPUs.
vLLM: Useful for micro-datacenter edge clusters with multiple GPUs, though its memory overhead is higher.

For containerized edge fleets, package these runtimes with Docker and orchestrate via K3s or similar lightweight Kubernetes distributions.

Hybrid Cloud-Edge Patterns

The most resilient architectures treat edge inference as a cache, not a replacement for cloud intelligence. Local models handle low-latency, high-frequency queries and PII-sensitive preprocessing. When the task requires long-context analysis, multi-step agentic reasoning, or large multimodal inputs, the edge node should forward the

Using LLMs for Autonomous Vehicles: A Practical Guide

shashank ms — Sun, 16 Aug 2026 07:35:42 +0000

Autonomous vehicle stacks need fast, interpretable reasoning over messy sensor data. In this guide, I will walk you through building a lightweight LLM agent that consumes structured scene descriptions and outputs validated driving decisions. We will run the entire pipeline against Oxlo.ai so you can prototype without managing inference infrastructure.

What you'll need

An Oxlo.ai API key from https://portal.oxlo.ai
Python 3.10 or newer
The OpenAI SDK: pip install openai

Step 1: Configure the Oxlo.ai client

We start by instantiating the OpenAI-compatible client pointing at Oxlo.ai. I keep the API key in an environment variable so it does not end up in source control.

import os
from openai import OpenAI

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key=os.environ.get("OXLO_API_KEY", "YOUR_OXLO_API_KEY")
)

Step 2: Define the AV system prompt

The system prompt constrains the model to behave like a safety-critical planner. It expects JSON sensor input and must return JSON containing a recommended action, confidence, and rationale. I treat this as a configurable constant so I can iterate quickly.

SYSTEM_PROMPT = """You are an autonomous vehicle decision agent. Your job is to analyze a structured scene snapshot and output a single JSON object with no markdown formatting.

Required JSON schema:
- action: one of [ACCELERATE, BRAKE, TURN_LEFT, TURN_RIGHT, MAINTAIN_SPEED, STOP]
- target_speed_mph: integer, 0 to 65
- confidence: float, 0.0 to 1.0
- rationale: string, max 200 characters
- hazard_detected: boolean

Rules:
1. Always prioritize pedestrian safety.
2. Respect traffic signals.
3. If occlusion is high and a pedestrian may be present, choose STOP or BRAKE.
4. Return only valid JSON. Do not include explanations outside the JSON."""

Step 3: Format sensor telemetry

Real AV pipelines publish object lists from perception. We simulate one tick of fused camera and lidar data, then serialize it into a concise text block for the model. Keeping the prompt compact reduces latency.

import json

def format_scene(
    ego_speed_mph: int,
    traffic_light: str,
    objects: list[dict],
    weather: str = "clear"
) -> str:
    scene = {
        "ego_speed_mph": ego_speed_mph,
        "traffic_light": traffic_light,
        "weather": weather,
        "detected_objects": objects
    }
    return json.dumps(scene, indent=2)

# Example tick
scene_message = format_scene(
    ego_speed_mph=25,
    traffic_light="green",
    objects=[
        {"type": "vehicle", "distance_m": 12, "lane": "same", "speed_mph": 20},
        {"type": "pedestrian", "distance_m": 8, "lane": "crosswalk", "status": "walking"}
    ],
    weather="fog"
)

Step 4: Query the model with JSON mode

We send the formatted scene to Oxlo.ai and request a JSON object back. I use llama-3.3-70b because it follows structured instructions reliably and has no cold starts on Oxlo.ai. If you need deeper reasoning for edge cases, swap in kimi-k2.6 or deepseek-v3.2 without changing any other code.

response = client.chat.completions.create(
    model="llama-3.3-70b",
    messages=[
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user", "content": scene_message},
    ],
    response_format={"type": "json_object"},
    temperature=0.1,
    max_tokens=256
)

raw_output = response.choices[0].message.content
decision = json.loads(raw_output)
print(json.dumps(decision, indent=2))

Step 5: Wrap the decision loop

For a real prototype, we need a reusable function that accepts a scene dict and returns a validated decision dict. I also add a small retry guard in case the model returns malformed JSON during early iterations.

def plan(scene_payload: str, max_retries: int = 2) -> dict:
    for attempt in range(max_retries):
        try:
            resp = client.chat.completions.create(
                model="llama-3.3-70b",
                messages=[
                    {"role": "system", "content": SYSTEM_PROMPT},
                    {"role": "user", "content": scene_payload},
                ],
                response_format={"type": "json_object"},
                temperature=0.1,
                max_tokens=256
            )
            content = resp.choices[0].message.content
            return json.loads(content)
        except Exception as e:
            if attempt == max_retries - 1:
                return {
                    "action": "STOP",
                    "target_speed_mph": 0,
                    "confidence": 1.0,
                    "rationale": f"Planner failed after {max_retries} attempts: {str(e)}",
                    "hazard_detected": True
                }
            continue

# Quick sanity check
test = plan(format_scene(
    ego_speed_mph=35,
    traffic_light="yellow",
    objects=[
        {"type": "vehicle", "distance_m": 30, "lane": "same", "speed_mph": 35},
        {"type": "pedestrian", "distance_m": 4, "lane": "crosswalk", "status": "walking"}
    ]
))
print(test)

Run it

Here is a complete script that simulates three consecutive perception ticks and prints the planner decisions. Because Oxlo.ai charges per request, not per token, feeding long object lists from lidar point clusters is predictable and cheap. See current rates at https://oxlo.ai/pricing.

if __name__ == "__main__":
    ticks = [
        format_scene(25, "green", [
            {"type": "vehicle", "distance_m": 15, "lane": "same", "speed_mph": 25}
        ]),
        format_scene(25, "red", [
            {"type": "vehicle", "distance_m": 5, "lane": "same", "speed_mph": 0}
        ]),
        format_scene(15, "green", [
            {"type": "pedestrian", "distance_m": 6, "lane": "crosswalk", "status": "standing"},
            {"type": "cyclist", "distance_m": 10, "lane": "right_adjacent", "speed_mph": 8}
        ], weather="rain")
    ]

    for i, tick in enumerate(ticks, 1):
        decision = plan(tick)
        print(f"Tick {i}: {decision['action']} - {decision['rationale']}")

Example output:

Tick 1: MAINTAIN_SPEED - Leading vehicle is moving at similar speed, green light, safe following distance.
Tick 2: BRAKE - Red light detected and lead vehicle stopped, decelerating to stop.
Tick 3: BRAKE - Pedestrian standing at crosswalk in rain requires caution, reducing speed.

Wrap up

You now have a working LLM decision layer for AV prototyping. Two concrete next steps: wire this planner into a ROS2 node so it consumes live perception topics, and add a memory buffer so the model sees the previous two ticks for temporal consistency. Both are straightforward because the Oxlo.ai endpoint is a drop-in OpenAI-compatible client, so you keep your existing Python async patterns.

Using LLM for Game Development: Tips and Techniques

shashank ms — Sun, 16 Aug 2026 05:35:40 +0000

Game development pipelines increasingly rely on large language models for everything from generating NPC dialogue trees to debugging shader scripts. The difference between a prototype and a polished feature often comes down to how quickly you can iterate on context-heavy prompts, agentic tool chains, and long-form narrative outputs. The infrastructure you choose needs to handle high-frequency, variable-length workloads without unpredictable costs.

NPC Dialogue and Dynamic Narrative

Consistent character voices require more than a one-shot prompt. Production workflows typically inject a character bible, prior conversation history, and world-state context into every request. These prompts can quickly span tens of thousands of tokens, especially when managing relationship graphs or faction allegiances across sessions.

Instead of trimming context to save money, structure your prompts into three blocks: a static personality matrix, a dynamic memory buffer of recent interactions, and a system instruction that constrains tone and formatting. Keep the full history in a vector store or graph database, then retrieve only the most relevant nodes for each turn. This approach preserves coherence without bloating every single request unnecessarily.

Procedural Content and World Building

LLMs excel at generating structured content like quests, item descriptions, and lore entries. The key is enforcing schema compliance so generated data slots directly into your game engine or CMS. Most modern inference APIs support JSON mode, which lets you define a Pydantic model or TypeScript interface and receive valid, parseable output.

A typical workflow looks like this: query your procedural generation seed, assemble a prompt that includes biome rules and rarity tables, and request a JSON array of loot objects. Validate the response against your schema before writing it to your content pipeline. If you are generating large batches, run requests in parallel and deduplicate entries with an embedding model to avoid repetitive flavor text.

Code Generation and Debugging

Game logic, scripting, and shader code are fertile ground for LLM assistance. Models tuned for code, such as Qwen 3 Coder 30B or DeepSeek Coder, handle Lua, Python, C#, and GLSL with high accuracy. For complex architectural decisions, reasoning models like DeepSeek R1 671B MoE or Kimi K2.6 provide step-by-step chain-of-thought outputs that help you audit the logic before it reaches your codebase.

When debugging, paste the error log, the suspect function, and a snippet of the surrounding class context. Ask the model to explain the failure mode and propose a minimal diff. Always run generated code in an isolated environment first, especially when dealing with engine-specific APIs that may have changed between versions.

Agentic Workflows for Game AI

Agentic systems turn an LLM from a text generator into an autonomous design assistant. By giving the model access to tools through function calling, you can let it query your asset database, adjust difficulty curves, or spawn test builds. These loops often involve multi-turn conversations with long system prompts and extensive tool definitions.

Because each turn carries the full conversation history plus tool schemas, token counts accumulate fast. A single agent session can easily process hundreds of thousands of tokens while tuning a boss encounter or balancing an economy table. This is where token-based billing creates friction, because the cost scales with every tool invocation and every line of context you retain.

Choosing the Right Model and Cost Structure

Game development workloads are uniquely demanding. You need fast iteration for dialogue, structured output for content pipelines, deep reasoning for algorithmic code, and sustained context for agentic loops. Oxlo.ai offers 45+ models across these exact categories, from the general-purpose Llama 3.3 70B and multilingual Qwen 3 32B to specialized options like DeepSeek R1 671B MoE for complex reasoning and Kimi K2.6 for agentic coding with vision support.

The bigger consideration is cost predictability. Token-based providers charge for every input and output token, which means long character bibles, code context windows, and multi-turn agent sessions drive up bills in direct proportion to your prompt length. Oxlo.ai uses flat per-request pricing: one fixed cost per API call regardless of how many tokens you send. For long-context and agentic game development workloads, this can make costs significantly more predictable and often far lower than token-based alternatives. See the exact rates on the Oxlo.ai pricing page.

Additional practical benefits matter for game dev teams. Oxlo.ai is fully OpenAI SDK compatible, so you can drop it into existing Python or Node.js tool chains without rewriting clients. There are no cold starts on popular models, which keeps iteration loops tight when you are rapidly testing dialogue variations or code refactors.

Getting Started with Oxlo.ai

The API is a direct replacement for the standard OpenAI client. Change the base URL and API key, and you can start generating content immediately. Below is a minimal Python example that generates a structured quest object using JSON mode.

import openai

client = openai.OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key="YOUR_OXLO_API_KEY"
)

response = client.chat.completions.create(
    model="llama-3.3-70b",
    messages=[
        {
            "role": "system",
            "content": "You are a senior narrative designer. Respond with valid JSON only."
        },
        {
            "role": "user",
            "content": (
                "Generate a side quest for a cyberpunk RPG. "
                "Include title, giver_name, objectives (array), and reward. "
                "Theme: corporate espionage. Difficulty: hard."
            )
        }
    ],
    response_format={"type": "json_object"}
)

import json
quest = json.loads(response.choices[0].message.content)
print(json.dumps(quest, indent=2))

For agentic prototypes, enable function calling and define your game tools as JSON schemas. Because Oxlo.ai bills per request, you can pass extensive tool definitions and maintain long conversation histories without watching token meters increment on every loop.

Whether you are building dynamic narrative systems, procedural content pipelines, or coding assistants for your team, the right inference backend removes cost surprises and keeps latency low. Oxlo.ai provides the model variety, SDK compatibility, and pricing structure that align with how game developers actually work.

Engineering Multimodal LLMs: Challenges and Solutions

shashank ms — Sun, 16 Aug 2026 03:38:55 +0000

We are building a multimodal incident diagnostician that consumes a Grafana screenshot and a raw log tail to produce a structured JSON root-cause analysis. It helps on-call engineers who are tired of context switching between dashboards and terminals at 3 a.m. The whole pipeline runs against Oxlo.ai's vision-capable models with flat per-request pricing, so adding a high-resolution image or extra log lines does not change the cost.

What you'll need

An Oxlo.ai API key from https://portal.oxlo.ai
Python 3.10 or newer
The OpenAI SDK: pip install openai

Step 1: Encode the screenshot

Vision models need images inlined as base64 data URLs. I wrote a small helper that detects the file extension and returns the proper RFC 2397 string. This keeps everything self-contained and avoids hosting images on a public CDN.

import base64
from pathlib import Path

def encode_image(path: str) -> str:
    ext = Path(path).suffix.lstrip(".")
    if ext == "jpg":
        ext = "jpeg"
    with open(path, "rb") as f:
        b64 = base64.b64encode(f.read()).decode("utf-8")
    return f"data:image/{ext};base64,{b64}"

if __name__ == "__main__":
    image_b64 = encode_image("dashboard.png")
    print(f"Encoded {len(image_b64)} characters")

Step 2: Lock down the system prompt

Hallucination is the biggest risk when a model interprets a chart. I lock the model into a rigid inspection protocol so it states "unreadable" instead of guessing. This prompt is the entire contract for the agent.

SYSTEM_PROMPT = """You are an on-call site reliability engineer. You have two inputs:
1. A screenshot of a Grafana dashboard panel.
2. A short tail of raw application logs.

Follow this protocol exactly:
- First, list every visible metric name and its approximate value in the screenshot.
- Second, correlate any spikes or anomalies with timestamped ERROR or FATAL lines in the logs.
- Third, emit a single JSON object with keys: "anomaly_seen" (bool), "metric" (str), "log_signature" (str), "root_cause" (str), "remediation" (str).

If a metric is unreadable, state "unreadable" rather than guessing."""

Step 3: Fire the multimodal request to Oxlo.ai

Kimi K2.6 handles vision, advanced reasoning, and a 131K context window, so we can stuff a large screenshot and verbose logs into one flat request. Because Oxlo.ai charges per request rather than per token, adding extra log lines or a high-resolution image does not inflate the cost. That predictability matters when you are iterating on prompts at 3 a.m. See https://oxlo.ai/pricing for current plan details.

from openai import OpenAI

client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")

user_message = [
    {"type": "image_url", "image_url": {"url": image_b64}},
    {"type": "text", "text": "Logs:\n2024-05-21T03:14:22Z ERROR connection pool exhausted\n2024-05-21T03:14:23Z FATAL request timeout after 30s\n2024-05-21T03:14:25Z ERROR retry failed"}
]

response = client.chat.completions.create(
    model="kimi-k2.6",
    messages=[
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user", "content": user_message},
    ],
    response_format={"type": "json_object"},
    temperature=0.2,
)

print(response.choices[0].message.content)

Step 4: Parse and validate the structured output

Raw JSON from a vision model often arrives wrapped in markdown fences. I strip those guards and parse the payload with the standard library so downstream automation can act on the result without string matching.

import json

raw = response.choices[0].message.content.strip()

if raw.startswith("

```"):
    raw = raw.split("\n", 1)[1].rsplit("```

", 1)[0].strip()

diag = json.loads(raw)
assert "anomaly_seen" in diag, "Missing anomaly_seen key"

print(f"Anomaly detected: {diag['anomaly_seen']}")
print(f"Root cause: {diag['root_cause']}")
print(f"Suggested fix: {diag['remediation']}")

Step 5: Bridge vision to action with tool use

A screenshot is only one frame. I give the model a fetch_logs function so it can request additional log ranges before finalizing its diagnosis. This bridges vision and action, which is the hard part of engineering multimodal agents.

import json

def fetch_logs(start: str, end: str) -> str:
    # Stub for your internal log store.
    return f"Stub logs from {start} to {end}"

tools = [
    {
        "type": "function",
        "function": {
            "name": "fetch_logs",
            "description": "Retrieve logs for a time range in ISO format.",
            "parameters": {
                "type": "object",
                "properties": {
                    "start": {"type": "string"},
                    "end": {"type": "string"}
                },
                "required": ["start", "end"]
            }
        }
    }
]

response = client.chat.completions.create(
    model="kimi-k2.6",
    messages=[
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user", "content": user_message},
    ],
    tools=tools,
    temperature=0.2,
)

msg = response.choices[0].message

if msg.tool_calls:
    tool_call = msg.tool_calls[0]
    args = json.loads(tool_call.function.arguments)
    extra_logs = fetch_logs(args["start"], args["end"])

    messages = [
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user", "content": user_message},
        {"role": "assistant", "content": msg.content or "", "tool_calls": [
            {"id": tool_call.id, "type": tool_call.type, "function": {"name": tool_call.function.name, "arguments": tool_call.function.arguments}}
        ]},
        {"role": "tool", "tool_call_id": tool_call.id, "content": extra_logs},
    ]

    final = client.chat.completions.create(
        model="kimi-k2.6",
        messages=messages,
        response_format={"type": "json_object"},
        temperature=0.2,
    )
    print(final.choices[0].message.content)
else:
    print(msg.content)

Run it

I combine the stable pieces into a single diagnose() function. Pass it a local PNG and a log tail, and it returns a validated dict.

import base64
import json
from pathlib import Path
from openai import OpenAI

def encode_image(path: str) -> str:
    ext = Path(path).suffix.lstrip(".")
    if ext == "jpg":
        ext = "jpeg"
    with open(path, "rb") as f:
        b64 = base64.b64encode(f.read()).decode("utf-8")
    return f"data:image/{ext};base64,{b64}"

SYSTEM_PROMPT = """You are an on-call site reliability engineer. You have two inputs:
1. A screenshot of a Grafana dashboard panel.
2. A short tail of raw application logs.

Follow this protocol exactly:
- First, list every visible metric name and its approximate value in the screenshot.
- Second, correlate any spikes or anomalies with timestamped ERROR or FATAL lines in the logs.
- Third, emit a single JSON object with keys: "anomaly_seen" (bool), "metric" (str), "log_signature" (str), "root_cause" (str), "remediation" (str).

If a metric is unreadable, state "unreadable" rather than guessing."""

def diagnose(image_path: str, log_tail: str) -> dict:
    image_b64 = encode_image(image_path)
    user_content = [
        {"type": "image_url", "image_url": {"url": image_b64}},
        {"type": "text", "text": f"Logs:\n{log_tail}"}
    ]

    client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")

    r = client.chat.completions.create(
        model="kimi-k2.6",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": user_content},
        ],
        response_format={"type": "json_object"},
        temperature=0.2,
    )

    raw = r.choices[0].message.content.strip()
    if raw.startswith("

```"):
        raw = raw.split("\n", 1)[1].rsplit("```

", 1)[0].strip()
    return json.loads(raw)

if __name__ == "__main__":
    logs = """2024-05-21T03:14:22Z ERROR connection pool exhausted
2024-05-21T03:14:23Z FATAL request timeout after 30s
2024-05-21T03:14:25Z ERROR retry failed"""
    result = diagnose("dashboard.png", logs)
    print(json.dumps(result, indent=2))

Example output:

{
  "anomaly_seen": true,
  "metric": "DB connection pool usage",
  "log_signature": "connection pool exhausted",
  "root_cause": "The connection pool maxed out at 03:14 UTC, causing cascading timeouts.",
  "remediation": "Increase pool size or add connection retry with exponential backoff."
}

Wrap-up

Wire this function into a PagerDuty webhook so incoming pages automatically trigger the diagnostician and post the JSON summary back to Slack. If latency becomes critical, swap Kimi K2.6 for Gemma 3 27B on Oxlo.ai; the request-based pricing means you can A/B test the trade-off between speed and reasoning depth without watching token meters spin. Check https://oxlo.ai/pricing to see which plan fits your volume.

Optimizing LLMs for Multimodal Learning: Best Practices

shashank ms — Sun, 16 Aug 2026 03:35:51 +0000

Multimodal LLMs process text, images, and audio in a single forward pass, but that convenience hides a sharp cost curve. Vision transformers encode images into hundreds or thousands of latent tokens, and under token-based pricing, a single high-resolution screenshot can cost more than a long text document. For production systems, optimizing how you prepare, route, and cache multimodal inputs is not an optional refinement. It is a necessity.

Understand Your Modality Mix

Text, image, and audio inputs are not priced equally on token-based platforms. A single image can expand into a thousand tokens depending on resolution, and audio segments add their own latent representations. Before you optimize, audit your traffic. Measure what percentage of your context window is consumed by each modality. If vision tokens dominate your spend, compression and caching should be your first targets. If audio is the driver, transcribing to text before reasoning is usually the efficient path.

Right-Size Vision Inputs

Most multimodal APIs accept arbitrary image resolutions, but the underlying vision encoder resamples them into a fixed grid of patches. A 1920x1080 screenshot might generate thousands of image tokens, while a 1024x1024 version of the same content often produces far fewer without a meaningful drop in comprehension.

Pre-processing images before they hit the API is the fastest way to cut costs. Resize, crop to the region of interest, strip metadata, and use efficient encoding. The following Python snippet uses Pillow to standardize inputs before base64 encoding them:

import base64
from io import BytesIO
from PIL import Image

def prepare_image(path, max_size=(1024, 1024), quality=85):
    img = Image.open(path).convert("RGB")
    img.thumbnail(max_size, Image.LANCZOS)
    buffer = BytesIO()
    img.save(buffer, format="JPEG", quality=quality)
    return base64.b64encode(buffer.getvalue()).decode("utf-8")

b64_image = prepare_image("dashboard.png")

Cache Reusable Visual Context

If your application repeatedly queries the same visual assets, such as UI mockups, documentation diagrams, or product catalogs, extract structured text descriptions once and reference them later. A small vision model can generate a detailed alt-text or JSON representation of an image. Subsequent reasoning steps can then run against cheaper text-only LLMs, avoiding the repeated token tax of resubmitting the image.

For conversational workflows that require true multimodal context, keep the image in the conversation history rather than re-uploading it every turn. This reduces bandwidth and, on token-based platforms, input token volume.

Route Tasks to Specialized Models

Not every vision task requires a frontier-scale model. Simple OCR, icon classification, or color extraction run well on smaller vision-language models. Complex reasoning over charts, cross-modal retrieval, or agentic coding loops benefit from larger checkpoints.

Oxlo.ai hosts multiple vision and general-purpose models on a single endpoint, including Gemma 3 27B and Kimi VL A3B for efficient vision tasks, and Kimi K2.6 or GLM 5 for advanced multimodal reasoning. Routing a lightweight vision job to a 27B parameter model instead of a 400B mixture-of-experts checkpoint can cut latency and cost without sacrificing accuracy for that specific task. Because Oxlo.ai is fully OpenAI SDK compatible, switching models is a one-line parameter change:

import openai

client = openai.OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key="YOUR_OXLO_API_KEY"
)

# Route simple vision tasks to a smaller Oxlo.ai model
response = client.chat.completions.create(
    model="gemma-3-27b-it",  # or kimi-vl-a3b for vision tasks
    messages=[{
        "role": "user",
        "content": [
            {"type": "text", "text": "List every button label in this UI."},
            {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{b64_image}"}}
        ]
    }]
)

Batch and Compress Multimodal Prompts

When you need to compare multiple images, arrange them into a single tiled grid rather than sending separate messages. A single composite image reduces the overhead of repeated system prompts and can lower the total number of image tokens, depending on the encoder. Similarly, combine text instructions so that one request handles extraction, classification, and formatting, using JSON mode to enforce structured output and eliminate follow-up calls.

Audio workloads follow the same logic. Chunk long recordings into semantically complete segments, transcribe them with Whisper, and feed the resulting text into a chat model. Running transcription and reasoning as discrete steps on specialized endpoints is usually cheaper than forcing a single large multimodal model to process raw audio for the entire pipeline.

Predictable Pricing for Vision Workloads

The biggest optimization is architectural. Token-based providers bill by total input and output tokens, which means a high-resolution image or a long audio clip can inflate costs unpredictably. For agentic systems that iteratively append screenshots, tool outputs, and conversation history, token counts compound quickly.

Oxlo.ai uses flat per-request pricing. One API call costs the same whether you send a short text prompt or a long-context multimodal payload with a high-resolution image and thousands of text tokens. For long-context workloads, request-based pricing can be 10-100x cheaper than token-based alternatives. For teams running agentic vision workflows, this removes the penalty for high-resolution inputs and makes costs predictable. You can budget by requests, not by tokens.

This pricing structure changes the optimization strategy. Instead of aggressively compressing every image to avoid token bloat, you can focus on accuracy and send the resolution the task actually requires. To see how request-based pricing fits your workload, visit https://oxlo.ai/pricing.

Conclusion

Multimodal optimization is a stack of small decisions. Resize images before encoding, cache visual context as structured text, route tasks to appropriately sized models, and batch related inputs into single requests. These practices keep latency low and quality high.

The final lever is your pricing model. If your application processes long documents, high-resolution images, or multi-turn agentic conversations, token-based scaling can dominate your budget. Oxlo.ai’s flat per-request pricing and broad multimodal catalog give you a predictable, developer-first platform for deploying vision, audio, and text workloads without the token tax.

LLMs for Speech Recognition: Challenges and Opportunities

shashank ms — Sun, 16 Aug 2026 03:34:51 +0000

Speech recognition has moved far beyond hidden Markov models and n-gram language models. Modern pipelines now combine large-scale speech encoders like Whisper with general-purpose LLMs to correct errors, format output, and reason over long-form audio. Yet deploying these systems at scale exposes practical challenges that token-based billing and fragmented model catalogs amplify. Audio is inherently high-entropy and lengthy, so transcription workloads often carry heavy context burdens and unpredictable costs.

The Current Landscape

OpenAI’s Whisper family remains the de facto foundation for open-source speech recognition. Researchers and engineers routinely pipe Whisper outputs into LLMs for post-processing, using the combined pipeline to restore punctuation, resolve homophones, and generate structured meeting notes. Concurrently, multimodal models are beginning to process audio directly, but for production workloads today, the two-stage approach, transcription followed by language model refinement, is the stable standard. The challenge is not model availability. It is operational friction: cold starts, incompatible SDKs, and pricing that balloons as transcripts grow.

Core Challenges

Hallucination and repetition. Whisper can hallucinate text during silence or repeat phrases in long-form audio. LLMs downstream must detect and strip these artifacts without stripping valid content.
Context length. A sixty-minute interview can produce fifteen thousand words. Feeding that into a model for summarization consumes substantial input context, which directly increases cost on token-based platforms.
Speaker diarization. Knowing who spoke when is still largely separate from transcription. Integrating diarization with LLM output requires careful prompt engineering and structured formats such as JSON mode.
Latency. Real-time agentic pipelines need streaming transcription and fast LLM turnaround. Cold starts on less popular speech models can break user experience.

Opportunities for LLM-Augmented ASR

LLMs open straightforward paths to higher-quality ASR output. A raw transcript can be passed through a reasoning model to standardize formatting, expand abbreviations, and correct domain-specific terminology. For developer workflows, LLMs can convert technical spoken content into executable code or structured logs. Multilingual pipelines benefit as well. Models such as Qwen 3 32B handle multilingual reasoning, so a transcript mixing English and Mandarin can be normalized into a single coherent language for downstream analytics. With function calling, the LLM can even route extracted action items directly into project management tools.

Building a Pipeline on Oxlo.ai

Oxlo.ai hosts Whisper Large v3, Whisper Turbo, and Whisper Medium alongside general-purpose and reasoning LLMs. Because the platform is fully OpenAI SDK compatible, you can transcribe audio and refine the result without switching clients or base URLs.

import openai

client = openai.OpenAI(
    api_key="YOUR_OXLO_API_KEY",
    base_url="https://api.oxlo.ai/v1"
)

# Stage 1: Transcribe
with open("meeting.wav", "rb") as audio_file:
    transcript = client.audio.transcriptions.create(
        model="whisper-large-v3",
        file=audio_file,
        response_format="text"
    )

# Stage 2: Format and summarize with an LLM
response = client.chat.completions.create(
    model="llama-3.3-70b",
    messages=[
        {"role": "system", "content": "Format the transcript. Add punctuation, fix spelling, and summarize key decisions."},
        {"role": "user", "content": transcript}
    ],
    temperature=0.1
)

print(response.choices[0].message.content)

With no cold starts on popular models, the pipeline is responsive. If you need deeper reasoning over technical content, you can swap llama-3.3-70b for deepseek-r1-671b or qwen-3-32b without changing any other infrastructure.

Why Request-Based Pricing Wins for Audio

Audio workloads are a natural stress test for token-based pricing. A ninety-minute podcast transcript sent to a model for chapterization can span tens of thousands of tokens. On token-based providers, the input cost for that single inference call scales linearly with transcript length. Oxlo.ai uses flat per-request pricing, so one API call costs the same regardless of whether the transcript is five hundred words or fifteen thousand. This makes long-context post-processing predictable and significantly cheaper for agentic speech workflows. For exact plan details, see the Oxlo.ai pricing page.

Conclusion

Speech recognition powered by LLMs is no longer experimental. It is a production requirement for meeting assistants, call analytics, and voice agents. The bottleneck has shifted from model accuracy to platform economics and integration friction. Oxlo.ai addresses both by offering Whisper and leading LLMs behind a single OpenAI-compatible endpoint, with request-based pricing that removes the penalty for long audio and long context. If you are building ASR pipelines that need to scale, Oxlo.ai is a relevant, cost-effective option to evaluate.