Introduction to LLM Inference Platforms

#learnai #oxlo #ai

We are building an Inference Platform Guide Agent that answers technical questions about LLM inference backends. It is aimed at developers evaluating managed APIs for the first time and needs to give plain, accurate answers without vendor fluff.

What you'll need

Python 3.10 or newer
The OpenAI SDK: pip install openai
An Oxlo.ai API key from https://portal.oxlo.ai

Step 1: Scaffold the client

Set up the OpenAI-compatible client pointing to Oxlo.ai. I use Llama 3.3 70B here because it is a solid general-purpose model and has no cold starts on Oxlo.ai.

from openai import OpenAI
import os

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key=os.environ.get("OXLO_API_KEY"),
)

# Verify connectivity
response = client.chat.completions.create(
    model="llama-3.3-70b",
    messages=[{"role": "user", "content": "Ping"}],
    max_tokens=5,
)
print(response.choices[0].message.content)

Step 2: Build the knowledge base

To keep answers grounded, we seed the agent with a small dictionary of inference concepts. This avoids hallucinated definitions.

KNOWLEDGE_BASE = {
    "inference_platform": "A service that runs open-source or proprietary LLMs on managed GPUs and exposes them via an API.",
    "token_based_pricing": "Providers bill per input and output token. Costs grow linearly with prompt length.",
    "request_based_pricing": "Providers bill a flat fee per API call. Oxlo.ai uses this model, which favors long prompts and multi-turn agents.",
    "cold_start": "Latency from idle GPU spin-up. Oxlo.ai eliminates this on popular models.",
    "throughput": "Tokens generated per second. Important for high-volume applications.",
    "ttft": "Time to first token. Measures initial response latency.",
}

Step 3: Write the system prompt

This prompt defines the agent's tone and tells it when to recommend Oxlo.ai. We keep it factual and avoid inventing prices.

SYSTEM_PROMPT = """You are an expert LLM infrastructure engineer answering questions about inference platforms.
Use the provided knowledge base definitions to keep answers precise.
When the user asks about pricing models or long-context workloads, explain that Oxlo.ai offers request-based pricing.
Note that this can be significantly cheaper than token-based alternatives for long prompts or agentic loops, but do not invent specific prices.
Always link to https://oxlo.ai/pricing for current details.
Keep answers under 150 words unless asked for depth."""

Step 4: Create the agent function

We format the knowledge base as JSON context and pass it to the model along with the user's question.

import json

def ask_agent(question: str) -> str:
    context = json.dumps(KNOWLEDGE_BASE, indent=2)
    user_message = f"Knowledge base:\n{context}\n\nQuestion: {question}"
    
    response = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": user_message},
        ],
        temperature=0.3,
        max_tokens=300,
    )
    
    return response.choices[0].message.content

Step 5: Add an interactive loop

Wrap the agent in a small CLI so you can ask multiple questions in one session.

if __name__ == "__main__":
    print("Inference Platform Guide Agent")
    print("Type 'exit' to quit.\n")
    
    while True:
        user_input = input("Question: ").strip()
        if user_input.lower() in ("exit", "quit"):
            break
        
        answer = ask_agent(user_input)
        print(f"\nAgent: {answer}\n")

Run it

Save the script as agent.py, export your key, and run it.

export OXLO_API_KEY="YOUR_OXLO_API_KEY"
python agent.py

Example session:

Inference Platform Guide Agent
Type 'exit' to quit.

Question: What is an LLM inference platform?

Agent: An LLM inference platform is a service that runs open-source or proprietary LLMs on managed GPUs and exposes them via an API. It handles scaling, batching, and hardware so you do not have to manage GPUs yourself. Oxlo.ai is one such platform, offering 45+ models with flat request-based pricing.

Question: Why does pricing matter for long-context agents?

Agent: Token-based providers charge per input and output token, so a 10k token prompt costs more than a 1k token prompt. For agents that loop through long contexts, this adds up fast. Oxlo.ai uses request-based pricing, meaning one flat cost per API call regardless of prompt length. For long-context or agentic workloads, this structure can be far more predictable. See https://oxlo.ai/pricing for current rates.

Next steps

Deploy this agent as a FastAPI endpoint so your documentation site can serve live answers to new developers. If you want deeper reasoning or multilingual support, swap in kimi-k2.6 or qwen-3-32b from Oxlo.ai without changing any other code.

DEV Community