Unlocking LLM Potential for Image Segmentation

Image segmentation has traditionally belonged to convolutional and diffusion-based architectures, but the reasoning capabilities of large language models are unlocking entirely new workflows. Instead of replacing U-Net or Mask R-CNN, modern vision-enabled LLMs act as orchestrators that interpret natural language, reason about spatial relationships, and drive specialized segmentation tools. Oxlo.ai provides the inference layer for these compound systems, offering vision models and general-purpose LLMs through a single OpenAI-compatible API. Because Oxlo.ai uses request-based pricing rather than token-based metering, long image descriptions and multi-turn refinement loops remain predictable even as prompts grow.

LLM-Driven Segmentation Patterns

There are three practical patterns for applying LLMs to segmentation tasks:

• Orchestration. The LLM receives an image plus a text prompt, then decides whether to invoke an object detection model, a SAM-based backend, or a custom CV pipeline.
• Coordinate generation. A vision-native LLM analyzes the image and outputs bounding boxes, point prompts, or polygon coordinates directly.
• Code synthesis. The LLM writes Python, OpenCV, or PyTorch code to process the image and extract masks.

Oxlo.ai supports all three approaches. The platform hosts vision models including Gemma 3 27B and Kimi VL A3B, alongside reasoning models such as Qwen 3 32B and Llama 3.3 70B, all exposed through the chat/completions endpoint with function calling, JSON mode, and streaming responses.

Orchestrating SAM with Vision LLMs

The most robust pattern treats the LLM as a planner. A vision model examines the image and produces structured coordinates or a tool call, while a dedicated segmentation model computes the final mask. Oxlo.ai also offers object detection models such as YOLOv9 and YOLOv11, which can bootstrap bounding boxes before mask refinement.

Below is a minimal Python example using the OpenAI SDK pointed at Oxlo.ai. The vision model inspects an image and emits a function call to generate_mask, passing target coordinates and a reasoning string.

import os
from openai import OpenAI

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key=os.environ["OXLO_API_KEY"]
)

tools = [
    {
        "type": "function",
        "function": {
            "name": "generate_mask",
            "description": "Run SAM on the given image using provided coordinates",
            "parameters": {
                "type": "object",
                "properties": {
                    "image_url": {"type": "string"},
                    "prompt_points": {
                        "type": "array",
                        "items": {"type": "number"},
                        "description": "Normalized x,y coordinates"
                    },
                    "reasoning": {"type": "string"}
                },
                "required": ["image_url", "prompt_points"]
            }
        }
    }
]

messages = [
    {
        "role": "user",
        "content": [
            {"type": "text", "text": "Segment the largest vehicle in this image."},
            {"type": "image_url", "image_url": {"url": "https://example.com/scene.jpg"}}
        ]
    }
]

response = client.chat.completions.create(
    model="gemma-3-27b",  # Also try Kimi VL A3B on Oxlo.ai
    messages=messages,
    tools=tools,
    tool_choice="auto"
)

print(response.choices[0].message.tool_calls)

The