Building Text Generation Tools with LLM: A Step-by-Step Guide

Text generation tools are the backbone of modern AI applications, from customer support bots to autonomous coding agents. Building these tools requires more than just a prompt and a model. You need a stack that handles streaming, structured output, function calling, and long-context workflows without unpredictable costs. This guide walks through constructing a production-ready text generation pipeline using the OpenAI SDK and an inference backend that treats long inputs as a feature, not a billing penalty.

Choosing the Right Model for Your Use Case

Selecting a model is the first architectural decision. Different tasks demand different capabilities, and Oxlo.ai offers 45+ open-source and proprietary models across seven categories.

• General reasoning and chat: Llama 3.3 70B and Qwen 3 32B handle multilingual dialogue and broad tasks.
• Deep reasoning and complex coding: DeepSeek R1 671B MoE and Kimi K2.6 excel at chain-of-thought reasoning, advanced coding, and vision tasks with 131K context.
• Efficient long-context: DeepSeek V4 Flash offers a 1M context window with efficient MoE architecture.
• Coding specialists: Qwen 3 Coder 30B, DeepSeek Coder, and Oxlo.ai Coder Fast are optimized for software generation.

Because Oxlo.ai hosts models for LLMs, code, vision, audio, and embeddings, you can route prompts to specialized endpoints without managing multiple provider accounts.

Project Setup and Authentication

Oxlo.ai is fully OpenAI SDK compatible, so you can use the official Python or Node.js client. Create an API key from the Oxlo.ai dashboard, then set your environment variables.

export OXLO_API_KEY="your-api-key"

Initialize the client with the Oxlo.ai base URL.

from openai import OpenAI
import os

client = OpenAI(
    api_key=os.getenv("OXLO_API_KEY"),
    base_url="https://api.oxlo.ai/v1"
)

Building a Basic Text Generation Pipeline

Start with a simple chat completion. Control creativity with temperature and set a max token limit to bound latency.

response = client.chat.completions.create(
    model="llama-3.3-70b",
    messages=[
        {"role": "system", "content": "You are a concise technical assistant."},
        {"role": "user", "content": "Explain the difference between token-based and request-based LLM pricing."}
    ],
    temperature=0.7,
    max_tokens=512
)

print(response.choices[0].message.content)

Streaming Responses for Real-Time UX

For interactive tools, blocking until the full response is generated creates a poor user experience. Enable streaming to emit tokens as they are produced.

stream = client.chat.completions.create(
    model="qwen3-32b",
    messages=[
        {"role": "user", "content": "Generate a step-by-step guide to deploying a FastAPI app to AWS."}
    ],
    stream=True
)

for chunk in stream:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)

Enforcing Structured Output with JSON Mode

Integrating LLM output into downstream systems often requires machine-readable structure. Use JSON mode to constrain the model to valid JSON.

response = client.chat.completions.create(
    model="deepseek-v3.2",
    messages=[
        {"role": "system", "content": "You are a JSON formatter. Return only valid JSON with keys: title, steps, tools_required."},
        {"role": "user", "content": "Plan a CI/CD pipeline for a Python microservice."}
    ],
    response_format={"type": "json_object"}
)

structured = response.choices[0].message.content

Adding Tool Use and Function Calling

Agents and automation tools need to interact with external APIs and code. Function calling lets the model decide when to invoke a registered tool and with what arguments.

tools = [
    {
        "type": "function",
        "function": {
            "name": "run_sql",
            "description": "Execute a read-only SQL query against the analytics database",
            "parameters": {
                "type": "object",
                "properties": {
                    "query": {"type": "string", "description": "The SQL query to run"}
                },
                "required": ["query"]
            }
        }
    }
]

response = client.chat.completions.create(
    model="kimi-k2.6",
    messages=[{"role": "user", "content": "How many signups did we have last Tuesday?"}],
    tools=tools,
    tool_choice="auto"
)

print(response.choices[0].message.tool_calls)

Handling Long-Context and Agentic Workloads

Retrieval-augmented generation and multi-step agents routinely send thousands or millions of tokens in a single request. Most token-based providers scale cost linearly with input length, which makes long-context prototyping and production deployment prohibitively expensive.

Oxlo.ai uses flat per-request pricing. A request costs the same whether it contains 100 tokens or 100,000 tokens. This makes Oxlo.ai significantly cheaper than token-based alternatives such as Together AI, Fireworks AI, OpenRouter, Replicate, and Anyscale when you are building RAG pipelines, agent loops, or document analysis tools. For workloads that require extreme context, DeepSeek V4 Flash supports a 1M context window, and Kimi K2.6 offers 131K context with advanced reasoning and vision capabilities.

You can explore the exact plan breakdown on the Oxlo.ai pricing page.

Cost Optimization and Predictable Billing

Unpredictable bills kill experimentation. With Oxlo.ai, you pay per request, not per token, so your monthly cost is a function of user interactions rather than prompt verbosity. The platform also offers no cold starts on popular models, which means consistent latency for user-facing tools.

The Free plan includes 60 requests per day across 16+ models, which is enough to prototype streaming, JSON mode, and tool use. When you are ready to scale, the Pro and Premium plans offer 1,000 and 5,000 requests per day respectively, with priority queue access under Premium. Enterprise teams can move to dedicated GPUs with custom volume pricing and a guaranteed reduction against their current provider bill.

Put