Integrating LLM into Web Applications with Long Context Workloads

#aiinfrastructure #oxlo #ai

Modern web applications increasingly rely on large language models to power chat interfaces, document analysis, and agentic workflows. As product requirements shift toward multi-turn conversations, large document ingestion, and persistent session memory, context windows have expanded from a few thousand tokens to hundreds of thousands or even millions. For engineering teams, this introduces a predictable cost problem. On token-based inference platforms, every additional paragraph in a prompt increases the bill. When building web apps that retain conversation history or process entire codebases, those increments compound quickly.

The Long Context Problem in Web Apps

Long-context workloads appear in almost every production LLM feature. A customer support widget that references ten prior messages, a legal SaaS tool that ingests a fifty-page contract, and a coding assistant that loads an entire repository into the prompt all share the same trait: the input is large, and it stays large across every request. On token-based providers such as Together AI, Fireworks AI, OpenRouter, Replicate, or Anyscale, pricing scales directly with the number of tokens in and out. This makes cost forecasting difficult and can discourage product teams from building richer, more contextual experiences.

Architecture Patterns for LLM Integration

Most production web applications do not call an LLM directly from the browser. Instead, they route requests through a backend service that handles authentication, prompt construction, and response streaming. A typical Node.js stack uses the OpenAI SDK with a custom base URL, which means you can swap providers without rewriting client code.

Oxlo.ai is fully OpenAI SDK compatible, so the integration looks identical to any other provider. You initialize the client, point it to https://api.oxlo.ai/v1, and use the same chat completions, embeddings, and image generation endpoints you already know.

Why Request Pricing Matters for Long Context

This is where Oxlo.ai diverges from the standard token-based model. Oxlo.ai uses flat per-request pricing: one cost per API call regardless of how many tokens are in the prompt. For long-context and agentic workloads, this can be 10-100x cheaper than token-based alternatives because cost does not scale with input length. You can pass an entire document, a full conversation log, or a lengthy system prompt, and the price remains the same.

Oxlo.ai offers 45+ models across seven categories, including long-context options such as DeepSeek V4 Flash with a 1 million token context window and Kimi K2.6 with 131K context. You can see the exact request-based rates on the Oxlo.ai pricing page.

Code Example: Streaming Chat Completion

The following Node.js example demonstrates a streaming chat endpoint using the OpenAI SDK pointed at Oxlo.ai. It is designed for a web client that sends a conversation history array. Because Oxlo.ai charges per request, you can keep the full history in context without worrying about token creep.

import OpenAI from "openai";

const openai = new OpenAI({
  apiKey: process.env.OXLO_AI_API_KEY,
  baseURL: "https://api.oxlo.ai/v1",
});

// Express route handler
app.post("/chat", async (req, res) => {
  const { messages } = req.body; // long multi-turn history is fine

  const stream = await openai.chat.completions.create({
    model: process.env.OXLO_AI_MODEL,
    messages: [
      { role: "system", content: "You are a helpful assistant." },
      ...messages,
    ],
    stream: true,
  });

  res.setHeader("Content-Type", "text/event-stream");
  res.setHeader("Cache-Control", "no-cache");
  res.setHeader("Connection", "keep-alive");

  for await (const chunk of stream) {
    const content = chunk.choices[0]?.delta?.content || "";
    res.write(`data: ${JSON.stringify({ content })}\n\n`);
  }

  res.write("data: [DONE]\n\n");
  res.end();
});

Notice that the messages array can contain dozens of prior turns or a full document preamble. Because Oxlo.ai does not meter by the token, your unit cost stays flat while the user experience remains stateful and rich.

Handling Large Documents and Memory

Document question and answer is one of the most common long-context patterns. Rather than chunking a PDF and running expensive retrieval pipelines, you can feed the entire extracted text into a single request when the model supports a large enough window. Oxlo.ai hosts models such as DeepSeek V4 Flash and Kimi K2.6 specifically for this use case. With no cold starts on popular models, the first request after idle time returns immediately, which is critical for interactive web apps.

Function Calling and Structured Output

Modern web apps rarely return raw text to the UI. They often trigger tools, update databases, or emit typed JSON. Oxlo.ai supports function calling, JSON mode, and vision inputs through the standard OpenAI SDK schema.

Below is an example that uses tool definitions to let the model decide whether to query an order database or respond directly. The input can still include a large system context or user history without affecting cost.

const tools = [
  {
    type: "function",
    function: {
      name: "get_order_status",
      description: "Retrieve the current status of a customer order.",
      parameters: {
        type: "object",
        properties: {
          order_id: { type: "string" },
        },
        required: ["order_id"],
      },
    },
  },
];

const response = await openai.chat.completions.create({
  model: process.env.OXLO_AI_MODEL,
  messages: [
    { role: "system", content: "You have access to the internal order system." },
    ...userMessages,
  ],
  tools,
  tool_choice: "auto",
});

const choice = response.choices[0];
if (choice.finish_reason === "tool_calls") {
  // Execute tool server-side and append result to conversation
}

Model Selection for Web Workloads

Oxlo.ai organizes its catalog into categories that map cleanly to web application needs. For general chat and reasoning, Llama 3.3 70B and Qwen 3 32B are solid defaults. For deep reasoning or complex coding, DeepSeek R1 671B MoE or GLM 5 provide advanced chain-of-thought capabilities. If you need to process images alongside text, Kimi VL A3B and Gemma 3 27B offer vision support. All of these are accessible through the same endpoint and SDK, so you can switch models by changing a single string.

Conclusion

Integrating LLMs into web applications is no longer just about calling a chat endpoint. It requires managing state, memory, documents, and tool use, all of which inflate prompt size. Token-based pricing punishes that richness. Oxlo.ai offers a developer-first alternative: flat per-request pricing, full OpenAI SDK compatibility, and a broad model catalog including long-context specialists like DeepSeek V4 Flash and Kimi K2.6. If your web app is moving beyond simple prompts into long-context territory, Oxlo.ai is a genuinely relevant option that keeps costs predictable as your features scale.

DEV Community