We are going to build a shelf-audit agent that ingests a retail shelf photo and returns a structured JSON inventory report plus a human-readable manager summary. It runs entirely on Oxlo.ai vision and chat models, so you get flat per-request pricing even when feeding large, high-resolution images. This is useful for retail operations teams that need to automate store compliance checks without building a custom CV pipeline.
What you'll need
- Python 3.10+
pip install openai- An Oxlo.ai API key from https://portal.oxlo.ai
- A sample shelf image named
shelf.jpg
Step 1: Configure the Oxlo.ai client
I start every project by verifying the client and credentials. This snippet points the OpenAI SDK at Oxlo.ai and makes a lightweight ping to confirm the key works.
from openai import OpenAI
import os
client = OpenAI(
base_url="https://api.oxlo.ai/v1",
api_key=os.getenv("OXLO_API_KEY")
)
# Verify connectivity with a cheap model
response = client.chat.completions.create(
model="deepseek-v3.2",
messages=[{"role": "user", "content": "ping"}],
max_tokens=5
)
print(response.choices[0].message.content)Step 2: Encode the input image
The chat completions vision format expects a base64 data URL. This helper reads any local image and returns the encoded string we need.
import base64
def encode_image(image_path):
with open(image_path, "rb") as image_file:
return base64.b64encode(image_file.read()).decode("utf-8")
image_b64 = encode_image("shelf.jpg")
print(f"Image encoded: {len(image_b64)} characters")Step 3: Write the auditor system prompt
A strict system prompt keeps the vision model from hallucinating products. I treat this as a contract, and I store it as a module-level constant so it versions cleanly with the codebase.
SYSTEM_PROMPT = """You are a retail shelf auditor. When a user provides a shelf image, analyze it and produce a strict JSON object with these keys:
- sku_count: integer, total visible product facings
- missing_skus: list of strings, expected products not visible
- compliance_score: integer 0-100
- restock_needed: boolean
- notes: list of strings, concise observations
Be precise. If you cannot see a product, do not hallucinate it."""Step 4: Run vision analysis with Kimi K2.6
Kimi K2.6 supports vision, reasoning, and a 131K context window, so it can handle high-resolution shelf photos and detailed instructions in one shot. I send the base64 image and enforce JSON mode so the output is machine readable.
user_message = {
"role": "user",
"content": [
{"type": "text", "text": "Analyze this shelf image and return the audit JSON."},
{
"type": "image_url",
"image_url": {"url": f"data:image/jpeg;base64,{image_b64}"}
}
]
}
response = client.chat.completions.create(
model="kimi-k2.6",
messages=[
{"role": "system", "content": SYSTEM_PROMPT},
user_message
],
response_format={"type": "json_object"},
max_tokens=1024
)
audit_json = response.choices[0].message.content
print(audit_json)Step 5: Parse and validate the audit JSON
I validate the structure immediately to catch schema drift before it hits downstream logic. This small guard saves production debugging time later.
import json
audit = json.loads(audit_json)
assert isinstance(audit.get("sku_count"), int)
assert isinstance(audit.get("compliance_score"), int)
print(f"SKUs: {audit['sku_count']}, Compliance: {audit['compliance_score']}/100")Step 6: Generate the manager summary with Qwen 3 32B
Raw JSON is not great for a store manager inbox, so I chain a text-only call to Qwen 3 32B to compress the audit into three plain-English bullets. Because Oxlo.ai uses flat request-based pricing, this follow-up does not inherit the image token cost from the vision step. You can see details at https://oxlo.ai/pricing.
summary_prompt = f"""Turn this shelf audit JSON into three bullet points for a store manager:
{json.dumps(audit, indent=2)}"""
summary_resp = client.chat.completions.create(
model="qwen-3-32b",
messages=[{"role": "user", "content": summary_prompt}],
max_tokens=256
)
summary = summary_resp.choices[0].message.content
print(summary)Run it
Here is the complete main block I run from the terminal. Below it is the actual output I captured from a test photo of a beverage aisle.
import os
import base64
import json
from openai import OpenAI
client = OpenAI(
base_url="https://api.oxlo.ai/v1",
api_key=os.getenv("OXLO_API_KEY")
)
SYSTEM_PROMPT = """You are a retail shelf auditor. When a user provides a shelf image, analyze it and produce a strict JSON object with these keys:
- sku_count: integer, total visible product facings
- missing_skus: list of strings, expected products not visible
- compliance_score: integer 0-100
- restock_needed: boolean
- notes: list of strings, concise observations
Be precise. If you cannot see a product, do not hallucinate it."""
def encode_image(image_path):
with open(image_path, "rb") as f:
return base64.b64encode(f.read()).decode("utf-8")
if __name__ == "__main__":
image_b64 = encode_image("shelf.jpg")
# Vision audit
vision_resp = client.chat.completions.create(
model="kimi-k2.6",
messages=[
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": [
{"type": "text", "text": "Analyze this shelf image and return the audit JSON."},
{"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{image_b64}"}}
]}
],
response_format={"type": "json_object"},
max_tokens=1024
)
audit = json.loads(vision_resp.choices[0].message.content)
print("=== AUDIT JSON ===")
print(json.dumps(audit, indent=2))
# Manager summary
summary_prompt = f"""Turn this shelf audit JSON into three bullet points for a store manager:
{json.dumps(audit, indent=2)}"""
summary_resp = client.chat.completions.create(
model="qwen-3-32b",
messages=[{"role": "user", "content": summary_prompt}],
max_tokens=256
)
print("\n=== MANAGER SUMMARY ===")
print(summary_resp.choices[0].message.content)=== AUDIT JSON ===
{
"sku_count": 24,
"missing_skus": ["Diet Cola 12-pack", "Sparkling Water Lime"],
"compliance_score": 78,
"restock_needed": true,
"notes": [
"Top shelf has two empty facings.",
"Brand blocking is broken on the second shelf.",
"Price tags are visible for all stocked items."
]
}
=== MANAGER SUMMARY ===
- We counted 24 facings but are missing Diet Cola 12-pack and Sparkling Water Lime.
- Compliance is at 78% because of empty space and broken brand blocking on the second shelf.
- Restock is needed immediately to close the gaps on the top shelf.
Wrap up
That is the core pipeline. To productionize it, expose the agent as a FastAPI endpoint so field staff can POST photos from a phone. You can also add a second vision pass with Gemma 3 27B to OCR price tags and cross-check shelf labels against the audit.
Top comments (0)