DEV Community

shashank ms
shashank ms

Posted on

Introduction to LLM Inference Platforms with Request-Based Pricing and Token-Based API

#learnai #oxlo #ai

I recently shipped an internal support tool that reads entire customer email threads and drafts first-pass replies. The agent runs on Oxlo.ai because our threads are long, and flat per-request pricing keeps the bill predictable regardless of input length. In this tutorial, you will build the same agent in Python.

What you'll need

Step 1: Configure the Oxlo.ai client

I started by wiring the OpenAI SDK to Oxlo.ai's compatible endpoint. No custom logic is needed. I defaulted to llama-3.3-70b for general reasoning, but you can swap in qwen-3-32b for multilingual tickets or kimi-k2.6 when you need a 131K context window.

import os
from openai import OpenAI

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key=os.environ["OXLO_API_KEY"]
)

MODEL = "llama-3.3-70b"

Step 2: Define the system prompt

I treat the system prompt as config. It sets the tone and constraints so the agent does not hallucinate policy details or overwhelm the customer.

SYSTEM_PROMPT = """You are a senior support engineer.
Read the full conversation thread below.
Identify the outstanding issue and draft a concise reply.
Rules:
- Acknowledge any delays if the thread shows frustration.
- Ask at most one clarifying question.
- Stay under 150 words.
- Never invent account specifics not present in the thread."""

Step 3: Load a long conversation thread

Support threads are noisy. I simulate a real workload by reading a text file containing thousands of words of back-and-forth. The length is the point: token-based platforms charge for every word in the input, but Oxlo.ai charges one flat request.

def load_thread(path):
    with open(path, "r", encoding="utf-8") as f:
        return f.read()

thread = load_thread("long_thread.txt")
print(f"Thread word count: {len(thread.split())}")

Step 4: Generate the reply

I send the entire thread in a single user message. Oxlo.ai handles the context without cold starts, and the cost is the same whether the thread is 500 words or 15,000 words. For extremely long logs, switch to deepseek-v4-flash and its 1M context window.

def draft_reply(text, model=MODEL):
    response = client.chat.completions.create(
        model=model,
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": f"Support thread:\n{text}\n\nDraft a reply."},
        ],
        temperature=0.3,
    )
    return response.choices[0].message.content

print(draft_reply(thread))

Step 5: Compare pricing models

My finance team wanted to know why I chose Oxlo.ai over our old token-based provider. I wrote a small helper that explains the structural difference. I do not hardcode competitor rates, but I show why per-request pricing is easier to forecast for long-context work.

def forecast_monthly(tickets_per_day):
    monthly_requests = tickets_per_day * 30
    return {
        "oxlo.ai": f"Oxlo.ai: {monthly_requests} requests. Flat per-request cost. See https://oxlo.ai/pricing",
        "token_based": f"Token-based: {monthly_requests} tickets, but total tokens vary with thread length.",
    }

print(forecast_monthly(50))

Run it

Create long_thread.txt. I generated mine by concatenating three real support exchanges until it hit about 4,000 words. Then run the script.

$ export OXLO_API_KEY="sk-..."
$ python support_agent.py

Thread word count: 4127
---
Thanks for your patience over the last six messages. I see the 502 error is still intermittent. To move forward, I need one piece of information: are you seeing this on every request or only during peak hours? I have attached the runbook for troubleshooting gateway timeouts in the meantime.
---
{'oxlo.ai': 'Oxlo.ai: 1500 requests. Flat per-request cost. See https://oxlo.ai/pricing', 'token_based': 'Token-based: 1500 tickets, but total tokens vary with thread length.'}

Wrap-up

That is the agent I run in production. Two concrete ways to extend it:

  1. Add function calling to query your CRM for account tier before drafting the reply. Oxlo.ai supports tool use on models like qwen-3-32b and llama-3.3-70b.
  2. Route complex technical threads to deepseek-r1-671b for deep reasoning, and keep simple tickets on deepseek-v3.2 to stay inside the free tier quota.

Top comments (0)