Leveraging LLMs for Personalized Recommendations: A Technical Deep Dive

Traditional recommendation systems rely on matrix factorization, two-tower embeddings, or gradient-boosted trees to map users to items. These approaches scale, but they struggle with cold-start items, sparse behavioral signals, and the need for explainable reasoning. Large language models offer an alternative. By encoding user history, item metadata, and situational context directly into the prompt, an LLM can perform zero-shot ranking, generate natural language explanations, and adapt to new domains without retraining. The challenge is not whether LLMs can recommend, but how to serve them cost-effectively at scale, especially when user histories grow long.

Why LLMs Change the Recommendation Stack

Classical recommenders embed users and items into a shared latent space. This works for head items, but falters when catalog metadata is rich and user intent is nuanced. LLMs invert the problem. They consume raw text: purchase logs, product descriptions, reviews, and real-time context. A model like Llama 3.3 70B or Qwen 3 32B can reason over this text to produce ranked lists, generate explanations, or even ask clarifying questions in a conversational flow.

The shift is architectural. Instead of maintaining separate feature stores, embedding indices, and re-ranking models, you can unify candidate selection and ranking inside a single context window. For agentic recommendation workflows, where the system iteratively refines criteria through tool use, the LLM becomes both the inference engine and the policy layer.

Retrieval-Augmented Generation for Recommendations

Raw LLM generation over an entire catalog of 100,000 items is impractical. The standard pattern is retrieval followed by ranking. First, retrieve a candidate set using an embedding model or a traditional ANN index. Then, pass the candidate metadata and user profile to the LLM for final ranking and explanation.

Oxlo.ai provides embedding models such as BGE-Large and E5-Large through the same OpenAI-compatible endpoint. You can generate user and item embeddings via the embeddings API, query your vector store, and feed the top-k results into a chat completion call. This two-stage design keeps latency low and quality high.

In practice, the prompt for the ranking stage contains:

1. A system prompt defining the recommendation policy.
2. A structured user history, for example, the last 50 interactions with timestamps and ratings.
3. The retrieved candidate set with titles, categories, and descriptions.
4. An instruction to output a ranked JSON list with reasoning.

Prompt Engineering and Context Windows

Recommendation prompts are inherently long. A detailed user profile with session logs, item descriptions, and few-shot examples can quickly consume tens of thousands of tokens. On token-based providers, this makes every request expensive, often prohibitively so for real-time personalization.

This is where Oxlo.ai's request-based pricing changes the economics. Because the cost is a flat rate per API call regardless of prompt length, long-context recommendation workloads do not trigger exponential cost growth. You can include full product descriptions, multi-turn conversation history, or long-term user behavior without token anxiety. For agentic loops that append tool results to a growing context, the predictability is significant.

Implementation with Oxlo.ai

Below is a minimal end-to-end example using the OpenAI Python SDK against Oxlo.ai. The example assumes you have already retrieved a candidate set via an embedding index.

import openai
import json

client = openai.OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key="YOUR_OXLO_API_KEY"
)

user_history = """
2024-01-15: Purchased "Advanced Python Programming"
2024-02-02: Viewed "Rust for Systems Engineers"
2024-02-10: Purchased "Designing Data-Intensive Applications"
"""

candidates = """
1. Title: "The Rust Programming Language", Category: Systems
2. Title: "Clean Architecture", Category: Software Engineering
3. Title: "Kubernetes in Action", Category: DevOps
"""

response = client.chat.completions.create(
    model="llama-3.3-70b",
    messages=[
        {
            "role": "system",
            "content": (
                "You are a technical book recommender. "
                "Rank the candidates by relevance to the user's history. "
                "Return strict JSON with keys: rank, title, reason."
            )
        },
        {
            "role": "user",
            "content": f"User History:\n{user_history}\n\nCandidates:\n{candidates}"
        }
    ],
    response_format={"type": "json_object"}
)

recommendations = json.loads(response.choices[0].message.content)
print(json.dumps(recommendations, indent=2))

The response_format={"type": "json_object"} flag ensures structured output, which lets you parse the result directly into your application layer without brittle regex extraction.

Structured Output and Tool Use

Production recommendation systems rarely return raw text. They return structured payloads that downstream services consume. Oxlo.ai supports JSON mode and function calling across its chat models, so you can define a schema for recommendations and enforce it at inference time.

For more advanced agentic behavior, you can register tools that the model can call to fetch real-time inventory, apply business rules, or look up user segments. Because Oxlo.ai offers no cold starts on popular models, these tool-augmented loops remain responsive even under variable load.

Cost Model for Long Context

The dominant cost driver in LLM-based recommendations is not the number of users, but the length of the context per user. A single personalized request can include thousands of tokens of history plus thousands more of catalog metadata. Under token-based billing, this multiplies costs linearly with context length.

Oxlo.ai uses request-based pricing: one flat cost per API request. For recommendation use cases, this can yield substantial savings on long-context and agentic workloads compared to token-based alternatives. Instead of compressing user history aggressively or pruning catalog descriptions to save tokens, you can send the full context required for high-quality ranking. Details are available on the Oxlo.ai pricing page.

Model Selection

Oxlo.ai hosts over 45 models across seven categories, all accessible through the same OpenAI-compatible endpoint. For recommendation workloads, the selection depends on latency and reasoning requirements.

• Llama 3.3 70B works well as a general-purpose ranking and explanation model.
• Qwen 3 32B offers strong multilingual reasoning if your catalog or user base spans languages.
• DeepSeek R1