Building a Recommender System using LLM

#aiinfrastructure #oxlo #ai

Large language models are increasingly used as the backbone of modern recommender systems, not just for generating text, but for encoding semantic meaning from unstructured catalogs and user histories. Unlike matrix factorization or shallow content-based filters, LLMs can ingest item descriptions, reviews, and interaction logs to produce nuanced representations that capture intent beyond simple co-occurrence. For teams building these pipelines, inference cost and latency become critical variables, especially when prompts grow to include verbose product details or lengthy user sessions. Oxlo.ai offers a developer-first inference platform with request-based pricing that keeps costs predictable regardless of input length, making it a practical choice for retrieval and ranking stages that process long-context data.

Embedding Retrieval with Oxlo.ai

The first stage of an LLM-powered recommender is typically semantic retrieval. Instead of relying solely on user-item interaction matrices, you encode item metadata and user profiles into dense vectors and search via approximate nearest neighbors. Oxlo.ai provides dedicated embedding endpoints through its OpenAI-compatible API, including models such as BGE-Large and E5-Large.

You can generate item embeddings in batches. Because Oxlo.ai charges per request rather than per token, sending a long product description with extensive metadata in a single API call does not inflate your bill. This is useful when catalog items include detailed specifications, HTML snippets, or multilingual text that would consume thousands of tokens under traditional pricing.

import openai

client = openai.OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key="YOUR_OXLO_API_KEY"
)

def embed_items(descriptions, model="bge-large"):
    response = client.embeddings.create(
        model=model,
        input=descriptions
    )
    return [e.embedding for e in response.data]

# Long product descriptions with metadata
items = [
    "Wireless noise-canceling headphones, 40hr battery, Bluetooth 5.3, ...",
    "Mechanical keyboard, hot-swappable switches, RGB backlight, ..."
]
vectors = embed_items(items)

Prompt-Based Re-Ranking

After retrieving candidates from a vector store, the next step is ranking. A lightweight but effective approach is to present the candidate items alongside the user's recent interactions in a structured prompt, then ask the LLM to return a relevance score or ordered list. Oxlo.ai supports JSON mode and function calling, so you can constrain the output to a machine-readable format without fragile regex parsing.

For this stage, models like Llama 3.3 70B or Qwen 3 32B work well for general reasoning, while DeepSeek R1 671B MoE or Kimi K2.6 can handle complex chain-of-thought reasoning if the ranking logic requires comparing nuanced attributes. Since user histories in production can easily span dozens of previous clicks or purchases, prompts become long. With Oxlo.ai's flat per-request pricing, you avoid the cost escalation that token-based providers impose on long-context workloads.

def rank_candidates(user_history, candidates, model="llama-3.3-70b"):
    system_msg = "You are a ranking assistant. Score each candidate 1-10."
    user_msg = f"User history: {user_history}\n\nCandidates:\n"
    for i, c in enumerate(candidates):
        user_msg += f"{i+1}. {c}\n"
    user_msg += "\nReturn JSON with keys: rankings (list of {id, score, reason})."

    response = client.chat.completions.create(
        model=model,
        messages=[
            {"role": "system", "content": system_msg},
            {"role": "user", "content": user_msg}
        ],
        response_format={"type": "json_object"},
        temperature=0.2
    )
    return response.choices[0].message.content

End-to-End Generation and Explanation

Some pipelines skip explicit retrieval altogether and use the LLM as a generative recommender, or they augment retrieval with natural language explanations. For example, the model can produce a personalized summary of why a specific item fits the user's taste. This requires a chat completion endpoint with strong instruction following.

Oxlo.ai offers 45+ models across seven categories, including vision and code specialists, though for text-based recommendation the LLM and chat category is most relevant. The platform is fully OpenAI SDK compatible, so switching from another provider requires only changing the base URL and API key. There are no cold starts on popular models, which matters for recommendation APIs that must respond within milliseconds during peak traffic.

Cost and Infrastructure Considerations

Recommender systems are inference-heavy. Every user event can trigger embedding updates, candidate retrieval, and ranking calls. When prompts include thousands of tokens of catalog data or session history, token-based bills scale linearly with context length. Most inference providers, including Together AI, Fireworks AI, OpenRouter, Replicate, and Anyscale, meter by token, which means long prompts directly increase cost. Oxlo.ai uses request-based pricing instead, so one flat cost per API request replaces variable token metering and can be significantly cheaper for long-context workloads.

For prototyping, the Oxlo.ai Free tier provides 60 requests per day across 16+ free models, including access to options like DeepSeek V3.2. When you move to production, Pro and Premium plans offer 1,000 and 5,000 requests per day respectively, with priority queue access under Premium. Enterprise plans add dedicated GPUs and a guaranteed 30% reduction off your current provider. See https://oxlo.ai/pricing for current plan details.

Because the platform exposes standard endpoints for chat, embeddings, and even image or audio generation, you can extend this architecture to multimodal catalogs without managing multiple SDKs.

Implementation Blueprint

A complete minimal pipeline looks like this:

Ingest item metadata and embed via the Oxlo.ai embeddings endpoint.
Index vectors in a store such as pgvector, Pinecone, or Weaviate.
On user action, embed the user profile or recent session text.
Retrieve top-K candidates via vector similarity.
Re-rank with an Oxlo.ai chat model using JSON mode, feeding in candidate details and user history.
Return the final list to the application layer.

This architecture separates concerns: embeddings handle broad retrieval efficiently, while the LLM handles nuanced ranking and explanation generation. Both stages benefit from predictable per-request pricing when input lengths vary.

Conclusion

Building recommender systems with LLMs gives you semantic flexibility that traditional approaches lack, but it also introduces inference cost uncertainty as prompts grow. Oxlo.ai addresses this with request-based pricing, an OpenAI-compatible API, and a broad model catalog that covers embedding, ranking, and generation needs. If you are prototyping a new recommendation pipeline or scaling one that processes long user sessions, Oxlo.ai provides a flat-cost infrastructure layer that stays predictable as your context windows expand.

DEV Community