Leveraging LLM for Recommender Systems: A Technical Deep Dive

Traditional recommender systems compress user behavior into latent vectors through matrix factorization or two-tower networks. This approach scales, but it sacrifices interpretability and struggles with cold-start items, sparse signals, and cross-domain reasoning. Large language models invert this trade-off. By consuming raw interaction histories, item metadata, and reviews as natural language, LLMs can reason explicitly about user preferences rather than approximating them through dot products. The challenge shifts from model architecture to inference architecture: how do you feed rich context into a model, control its output, and do it at a cost that survives production traffic?

Prompt-Based Ranking and the Context Budget

The simplest integration treats recommendation as a ranking task. You construct a prompt that contains a user profile, a compressed interaction history, and a slate of candidate items with metadata. The model returns an ordered list or relevance scores.

This works best when the candidate set is small, typically under 50 items, and when item descriptions are semantically rich. The bottleneck is context length. A detailed user history with timestamps, ratings, and review snippets can consume thousands of tokens before a single candidate is introduced.

This is where inference pricing structure directly impacts architecture. Under token-based billing, long user histories and detailed metadata incur escalating costs. Oxlo.ai uses flat per-request pricing, so expanding the prompt to include richer context does not inflate the inference cost. For recommender workloads, where prompts are inherently long, this can change the unit economics significantly. See Oxlo.ai pricing for plan details.

from openai import OpenAI
import json, os

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key=os.environ["OXLO_API_KEY"]
)

def rank_candidates(user_history: str, candidates: list[dict]) -> list[dict]:
    candidate_text = "\n".join(
        f"{i+1}. {c['title']} | {c['genre']} | {c['synopsis']}"
        for i, c in enumerate(candidates)
    )

    response = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=[
            {
                "role": "system",
                "content": "You are a recommendation engine. Rank items by predicted user preference. Return JSON."
            },
            {
                "role": "user",
                "content": f"User history:\n{user_history}\n\nCandidates:\n{candidate_text}\n\nReturn a JSON array of objects with keys id, rank, and reason."
            }
        ],
        response_format={"type": "json_object"},
        temperature=0.2,
    )
    return json.loads(response.choices[0].message.content)

Using response_format enforces structured output, which is critical for downstream consumption. Oxlo.ai supports JSON mode across its chat models, including Llama 3.3 70B and Qwen 3 32B.

Retrieval-Augmented Generation at Inference Time

For catalogs beyond a few dozen items, in-context ranking is impractical. The standard pattern is retrieve-then-rerank. First, a retrieval stage narrows the catalog to a top-k candidate set using embeddings or BM25. Then an LLM reranks this shortlist using nuanced reasoning.

Embeddings for retrieval can be generated through Oxlo.ai's embedding endpoints. Models like BGE-Large and E5-Large are available for this stage. The rerank stage benefits from reasoning-capable models such as DeepSeek R1 671B MoE or Kimi K2.6, which can compare trade-offs across item attributes explicitly.

# Stage 1: embed user profile and items
embed_response = client.embeddings.create(
    model="bge-large",
    input=[user_profile_text] + [item["description"] for item in catalog]
)

# Retrieve top-k via cosine similarity
# ... standard vector search ...

# Stage 2: LLM rerank
rerank_response = client.chat.completions.create(
    model="deepseek-r1-671b",
    messages=[{
        "role": "user",
        "content": f"Rerank these {len(top_k)} items for the user based on their history and preferences. Return JSON."
    }],
    response_format={"type": "json_object"}
)

Because the rerank prompt includes the user history plus full item descriptions, it is often longer than typical chat prompts. Flat per-request pricing removes the penalty for including rich metadata that improves relevance.

Agentic Recommendations with Tool Use

The most expressive pattern treats the LLM as an agent that orchestrates a multi-step recommendation workflow. Instead of a single prompt, the model calls tools to fetch fresh user data, query inventory, filter by business rules, and explain its choices.

Oxlo.ai supports function calling across its chat models, which allows the recommender to interact with external APIs without hardcoded orchestration logic.

tools = [

    {

        "type": "function",

        "function": {

            "name": "get_recent_interactions",

            "description": "Fetch user's last 20 interactions",

            "parameters": {

                "type": "