Building Vision-Based Chatbots with LLM

Vision-language models have moved chatbots beyond pure text. Users can now upload screenshots, diagrams, or photos and ask questions about them. For developers, this means building richer agents, but it also introduces complexity around context windows, image encoding, and unpredictable costs. This guide shows how to build a robust vision-based chatbot using an OpenAI-compatible API, and it explains why request-based inference changes the economics of image-heavy workloads.

What Makes Vision-Based Chatbots Different?

A standard text chatbot processes strings. A vision chatbot processes interleaved text and image data. Most modern APIs accept image inputs as base64-encoded strings or public URLs alongside text in the same message payload. The model then processes the combined context to generate a response.

This multimodal approach creates two engineering challenges. First, images consume large amounts of context window once tokenized, which can exhaust limits during multi-turn conversations. Second, pricing on token-based platforms scales with every image pixel that enters the context, making high-resolution or multi-image workflows expensive to run. A request-based pricing model removes that variable.

Choosing a Vision-Capable Model

Not every LLM accepts image input. You need a vision-language model (VLM) that supports interleaved content in the chat completions endpoint.

On Oxlo.ai, the vision category includes Gemma 3 27B and Kimi VL A3B for dedicated vision tasks. If you need advanced reasoning combined with image understanding, Kimi K2.6 supports vision alongside agentic coding and a 131K context window. Oxlo.ai hosts 45+ open-source and proprietary models across 7 categories, including these vision options, with no cold starts on popular models. Because the platform is fully OpenAI SDK compatible, you can point your existing client at the Oxlo.ai endpoint and call any of these models without rewriting your stack.

Setting Up the Oxlo.ai API

Oxlo.ai is a drop-in replacement for the OpenAI SDK. Set the base URL to https://api.oxlo.ai/v1 and use your Oxlo.ai API key.

from openai import OpenAI

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key="YOUR_OXLO_API_KEY"
)

MODEL = "gemma-3-27b-it"  # or kimi-vl-a3b, kimi-k2-6, etc.

This compatibility extends to Python, Node.js, and cURL. The chat.completions endpoint, streaming responses, and tool-calling schemas all work identically.

Building the Core Chat Loop

A vision chatbot is still a chatbot. Maintain a message history list and append user and assistant turns. The only difference is that a user message can now contain a list of content parts instead of a single text string.

messages = [
    {
        "role": "system",
        "content": "You are a helpful assistant that can analyze images."
    }
]

def chat(user_text, image_b64=None):
    content = [{"type": "text", "text": user_text}]
    
    if image_b64:
        content.append({
            "type": "image_url",
            "image_url": {
                "url": f"data:image/jpeg;base64,{image_b64}"
            }
        })
    
    messages.append({"role": "user", "content": content})
    
    response = client.chat.completions.create(
        model=MODEL,
        messages=messages,
        stream=True
    )
    
    reply = ""
    for chunk in response:
        delta = chunk.choices[0].delta.content or ""
        reply += delta
        print(delta, end="", flush=True)
    
    messages.append({"role": "assistant", "content": reply})
    return reply

This pattern supports multi-turn conversations where images and text are mixed across the thread. Streaming keeps the interface responsive even when the model is processing a large visual context.

Handling Image Inputs

Before sending an image, convert it to a base64 string. Keep the resolution reasonable. Very large images increase latency and context usage without always improving model accuracy.

import base64

def encode_image(path):
    with open(path, "rb") as f:
        return base64.b64encode(f.read()).decode("utf-8")

image_b64 = encode_image("diagram.png")
chat("Explain this architecture diagram.", image_b64=image_b64)

If you are building a web interface, you can accept a file upload, resize it server-side, encode it, and pass it into the message payload. Oxlo.ai supports vision inputs through the same chat/completions endpoint you use for text, so no additional routing logic is required.

Managing Context and Cost

Vision workloads are inherently long-context workloads. A single high-resolution image can represent thousands of text tokens once processed by the model. On token-based providers, that means every image upload directly increases your bill.

Oxlo.ai uses request-based pricing. You pay one flat cost per API request regardless of prompt length. Unlike token-based providers, cost does not scale with input length, so Oxlo.ai is significantly cheaper for long-context and agentic workloads that pass multiple images or large screenshots through the conversation thread. For exact plan details, see the Oxlo.ai pricing page.

To manage context window limits, consider these strategies:

• Resize images to the minimum resolution that preserves the task detail.
• Summarize older turns and drop raw image data from history once it is no longer needed.
• Use models with larger context windows, such as Kimi K2.6 with 131K context, when you must retain full image history.

Production Considerations

Once the core loop works, add controls for reliability and structure.

Structured output. Use JSON mode when you need the model to return parseable data, such as bounding box coordinates or structured annotations of an image.

Tool use. Function calling lets the chatbot invoke external tools based on image content. For example, the model could analyze a receipt image and then call a tool to log the expense.

Error handling. Vision models occasionally refuse to answer or hallucinate details. Implement retry logic and validate outputs before acting on them.

Model selection. Gemma 3 27B is a strong general-purpose vision model. For tasks that mix deep reasoning with visual input, Kimi K2.6 or Kimi VL A3B may be more appropriate. Oxlo.ai offers 45+ models, so you can route to different endpoints based on task complexity without managing separate provider accounts.

Conclusion

Building a vision-based chatbot does not require a new stack. If you already use the OpenAI SDK, you can add image handling to your existing chat.completions logic and switch the base URL to Oxlo.ai. With vision models like Gemma 3 27B, Kimi VL A3B, and Kimi K2.6, plus request-based pricing that stays flat even when users upload large screenshots or multi-image prompts, Oxlo.ai is a relevant option for developers shipping multimodal agents.