Recommender systems have historically relied on matrix factorization, two-tower embeddings, and gradient-boosted decision trees. These methods excel at scale but struggle with cold-start items, cross-domain reasoning, and explaining why an item fits a user's taste. Large language models introduce a different capability: they can interpret unstructured item metadata, reason about user intent from natural language, and generate contextual explanations in real time.
From Embeddings to Reasoning
Traditional collaborative filtering represents users and items as dense vectors. This works well when behavioral data is abundant, but it treats item content as a side channel. LLMs invert this assumption. By feeding item titles, descriptions, categories, and reviews directly into the prompt, the model reasons over the raw text rather than a compressed embedding. This is especially useful for cold-start scenarios where interaction history is sparse but catalog metadata is rich.
LLMs also unify tasks that previously required separate pipelines. A single model can rank candidates, generate explanations, and answer follow-up questions like Why not the cheaper option? or Show me something similar but shorter. This reduces system complexity and removes the need to synchronize multiple specialized models.
Architectural Patterns for LLM Recommendations
Most production deployments do not replace existing retrieval infrastructure. Instead, they layer an LLM on top of a candidate generator. The two dominant patterns are retrieval-augmented ranking and tool-augmented agents.
In retrieval-augmented ranking, a vector database or approximate nearest neighbor index retrieves 50 to 200 candidate items. The LLM receives the user profile, recent interactions, and candidate metadata, then returns a ranked subset with justifications. This keeps latency manageable because the LLM only scores a small basket rather than the full catalog.
In tool-augmented agents, the LLM is given function definitions that query structured databases, filter by inventory constraints, or check pricing rules. The model calls these tools iteratively, then synthesizes a final recommendation list. This pattern naturally fits agentic workflows and benefits from long context windows to maintain state across turns.
A Concrete Example: Prompt-Based Ranking
Below is a minimal example using the OpenAI SDK with Oxlo.ai to rerank movie recommendations. The prompt includes a short user history and a set of candidate films. Because Oxlo.ai uses request-based pricing, the cost is flat even if we include long plot summaries or few-shot examples in the prompt.
import openai
client = openai.OpenAI(
base_url="https://api.oxlo.ai/v1",
api_key="YOUR_OXLO_API_KEY"
)
user_history = "Recently watched: Blade Runner 2049, Interstellar, Arrival."
candidates = [
{"id": "tt0137523", "title": "Fight Club", "plot": "An insomniac office worker forms an underground fight club."},
{"id": "tt1375666", "title": "Inception", "plot": "A thief who steals corporate secrets through dream-sharing technology."},
{"id": "tt0816692", "title": "Interstellar", "plot": "A team of explorers travel through a wormhole in space."}
]
prompt = f"""You are a movie recommender. Given the user's watch history and candidate films, return the top 2 recommendations in JSON format.
User history: {user_history}
Candidates:
""" + "\n".join(f'- {c["title"]}: {c["plot"]}' for c in candidates) + """
Respond with a JSON array of objects with fields: id, title, reason.
"""
response = client.chat.completions.create(
model="qwen3-32b",
messages=[{"role": "user", "content": prompt}],
response_format={"type": "json_object"}
)
print(response.choices[0].message.content)
The example uses qwen3-32b, a strong multilingual reasoning model available on Oxlo.ai. You can swap it for llama-3.3-70b or deepseek-r1-671b depending on whether you need general-purpose fluency or deep chain-of-thought reasoning. Because Oxlo.ai is fully OpenAI SDK compatible, the only change required is the base_url.
Why Inference Economics Matter for Recommenders
Recommendation prompts are often longer than typical chat prompts. A single request can include a user profile, dozens of candidate item descriptions, few-shot examples, and system instructions. Under token-based pricing, every additional item description increases cost. For systems that rerank hundreds of candidates per user or maintain long conversational sessions, this scales linearly with catalog depth and session length.
Oxlo.ai uses request-based pricing: one flat cost per API request regardless of prompt length. For long-context and agentic recommendation workloads, this can be significantly more efficient than token-based alternatives such as Together AI, Fireworks AI, OpenRouter, Replicate, or Anyscale. You can include richer metadata, longer histories, and more candidates without watching inference costs scale proportionally. See https://oxlo.ai/pricing for current plan details.
Model Selection on Oxlo.ai
Oxlo.ai hosts over 45 models across seven categories, all accessible through the same OpenAI-compatible endpoint. For recommender systems, the following are particularly relevant:
- Qwen 3 32B and Llama 3.3 70B: Ideal for general ranking, explanation generation, and multilingual catalogs.
- DeepSeek R1 671B MoE and Kimi K2.6: Use these when you need advanced reasoning over complex user constraints or long interaction histories. Kimi K2.6 supports 131K context, and DeepSeek V4 Flash supports up to 1M tokens for feeding large candidate sets.
- BGE-Large and E5-Large: Use these embedding models to build the retrieval index that feeds candidates into the LLM reranker.
All models run with no cold starts, so latency is predictable even for sporadic recommendation traffic.
Practical Considerations
LLMs are not a wholesale replacement for matrix factorization at billion-item scale. They are best used as a reranking or explanation layer, or as a conversational interface on top of a traditional retrieval stack. Cache aggressively: user profiles and catalog embeddings change slowly, but LLM outputs can be cached for identical candidate baskets. When possible, batch multiple ranking decisions into a single prompt to amortize the request cost.
Hybrid systems often perform best. Use embeddings for fast candidate retrieval, then apply an LLM for nuanced ranking and natural language explanations. This preserves the scalability of classical recommenders while adding the interpretability and flexibility that LLMs provide.
Conclusion
Language models are reshaping recommender systems by enabling reasoning over unstructured metadata, conversational interaction, and transparent explanations. The shift from narrow embedding-based models to general reasoning engines changes not only architecture but also cost dynamics. Because recommendation workloads frequently involve long prompts, request-based pricing from Oxlo.ai offers a structurally different cost profile than token-based providers. With a broad model catalog, OpenAI SDK compatibility, and no cold starts, Oxlo.ai is a practical inference backend for teams building the next generation of recommenders.
Top comments (0)