Using LLM for Recommender Systems

#aiinfrastructure #oxlo #ai

Traditional recommender systems rely on matrix factorization, two-tower architectures, and densely engineered tabular features. While these methods scale efficiently to billions of interactions, they often struggle with cold-start items, cross-domain semantics, and generating human-readable explanations. Large language models introduce a different contract. They reason directly over unstructured text, infer nuanced preferences from raw user histories, and produce interpretable recommendations without extensive feature engineering.

Three Roles for LLMs in Recommendation

Most production pipelines that use LLMs fall into one of three patterns.

Feature extractor: Embedding models such as BGE-Large convert items and user profiles into dense vectors. These vectors feed a traditional nearest-neighbor index for candidate retrieval.
Scoring and ranking: The LLM receives a prompt containing the user history, candidate item descriptions, and explicit constraints. It returns a relevance score or a ranked list, often in JSON mode for structured parsing.
Generative and agentic recommender: The model acts as an agent. It asks clarifying questions, calls inventory or pricing tools via function calling, and iterates over multiple turns to refine a shortlist.

Each pattern benefits from long context windows. A rich user profile, complete with session logs, product reviews, and catalog metadata, can consume thousands of tokens. The cost model of your inference provider therefore becomes a core architectural constraint.

Long Context and Cost Predictability

Real-world recommendation prompts grow quickly. Passing a user’s six-month interaction history, detailed product descriptions, and few-shot examples easily exceeds standard context limits and inflates costs on token-based platforms. When cost scales linearly with input length, every additional behavioral signal carries a price penalty.

Oxlo.ai uses request-based pricing. You pay one flat cost per API request regardless of prompt length. For recommender systems that pass long interaction histories or large candidate sets in a single prompt, this structure removes the linear cost penalty associated with token-based providers. The result is predictable spend that does not scale with input size, which makes Oxlo.ai significantly cheaper for long-context and agentic workloads. See https://oxlo.ai/pricing for current plan details.

Implementation: Embedding Retrieval with LLM Reranking

A practical hybrid architecture uses an embedding model to retrieve candidates, then an LLM to rerank them using raw user history and natural language criteria. Oxlo.ai provides fully OpenAI SDK compatible endpoints for both stages.

import openai

client = openai.OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key="YOUR_API_KEY"
)

# 1. Generate candidate embeddings with BGE-Large
items = [
    {"id": "sku_42", "text": "Waterproof trail running shoes, aggressive lug pattern"},
    {"id": "sku_77", "text": "Minimalist road running shoe, lightweight mesh upper"},
    # ... additional catalog items
]

embedding_resp = client.embeddings.create(
    model="bge-large",  # BGE-Large on Oxlo.ai
    input=[i["text"] for i in items]
)

# 2. Retrieve top-k via vector similarity (pseudo-code)
# candidates = vector_db.query(embedding_resp.data, top_k=5)

# 3. Rerank with an LLM using the full user history
user_history = """
Previously purchased: men's wool base layer, size L.
Recent views: ultralight backpacks, trekking poles.
Returns: none.
"""

rerank_prompt = f"""
User profile:
{user_history}

Candidate items:
1. {items[0]['text']}
2. {items[1]['text']}

Rank the items by relevance to the user. Return JSON with id and score keys.
"""

rerank_resp = client.chat.completions.create(
    model="llama-3.3-70b",  # Llama 3.3 70B on Oxlo.ai
    messages=[{"role": "user", "content": rerank_prompt}],
    response_format={"type": "json_object"}
)

print(rerank_resp.choices[0].message.content)

This example uses bge-large for retrieval and llama-3.3-70b for ranking, both available on Oxlo.ai. Because the prompt contains the full user history, its token count is high. On Oxlo.ai, the cost remains a single request charge.

Agentic Recommendation Loops

For complex scenarios, such as travel planning or enterprise software procurement, a single prompt is insufficient. The model must ask clarifying questions, call external APIs for inventory or pricing, and iterate. This requires function calling, multi-turn conversation support, and streaming for latency-sensitive interfaces.

Oxlo.ai supports function calling, tool use, multi-turn conversations, and streaming responses. Models such as Qwen 3 32B and GLM 5 handle multilingual reasoning and long-horizon agentic tasks. You can build a recommendation agent that queries a product database, filters by availability, and explains trade-offs to the user across multiple turns without worrying about ballooning token costs per turn.

Model Selection on Oxlo.ai

Oxlo.ai hosts 45+ models across categories relevant to recommender pipelines.

Embeddings: BGE-Large and E5-Large for candidate retrieval and semantic similarity.
General reasoning: Llama 3.3 70B for robust ranking and preference understanding.
Multilingual and agentic: Qwen 3 32B for global catalogs and tool-use workflows.
Deep reasoning: DeepSeek R1 671B MoE and Kimi K2.6 for complex constraint satisfaction, such as budget-aware bundle recommendations.
Vision: Gemma 3 27B and Kimi VL A3B when recommendations depend on product imagery or user-generated photos.

DEV Community