Building a Recommender System using LLM and Collaborative Filtering

Recommender systems that blend collaborative signals with large language model reasoning are becoming the default for high-stakes retrieval and ranking. Collaborative filtering excels at surfacing behavioral patterns, but it struggles with cold-start items and sparse user histories. LLMs can close that gap by interpreting item metadata, user reviews, and session context at ranking time. The practical barrier is cost: when you pass long user histories and rich product descriptions to a model, token-based billing turns every additional context item into a direct expense. Oxlo.ai removes that constraint with flat per-request pricing, so you can build hybrid recommenders without trimming context to save tokens.

Architecture Overview

A production-grade hybrid recommender usually separates retrieval from ranking. In the retrieval stage, collaborative filtering narrows a catalog of millions down to a manageable candidate set. In the ranking stage, an LLM reorders those candidates by reasoning over the user’s full history, item attributes, and business rules. This two-stage design keeps latency low while letting the model focus its compute on the most relevant items.

Oxlo.ai hosts the open-source models you need for both stages. You can generate embeddings via BGE-Large or E5-Large for vector retrieval, then route the final ranking prompt to Llama 3.3 70B, DeepSeek V3.2, or Qwen 3 32B. Because Oxlo.ai is fully OpenAI SDK compatible, you can point your existing client at https://api.oxlo.ai/v1 and run the same code you would use against any OpenAI-compatible provider.

Collaborative Filtering for Candidate Generation

Start with a matrix factorization model to produce a baseline candidate set. The implicit library makes this straightforward on a user-item interaction matrix.

import numpy as np
from implicit.als import AlternatingLeastSquares
import scipy.sparse as sparse

# interactions: scipy CSR matrix of shape (n_users, n_items)
model = AlternatingLeastSquares(factors=128, regularization=0.05, iterations=20)
model.fit(interactions)

# For user_id, retrieve top 100 candidates
user_factors = model.user_factors
item_factors = model.item_factors

scores = item_factors.dot(user_factors[user_id])
candidate_indices = np.argpartition(scores, -100)[-100:]

These 100 candidates are a coarse filter. They capture behavioral similarity but contain no textual understanding of why one item might be preferred over another. That is the gap we fill with the LLM ranking layer.

Ranking Candidates with an LLM

For each user, construct a prompt that includes their recent interactions and a structured description of each candidate item. Ask the model to return a relevance score or a reordered list. This is where context length becomes a practical concern: a user with fifty past interactions and one hundred candidate items can easily consume tens of thousands of tokens in a single prompt.

On token-based platforms, that input length directly multiplies your cost. Oxlo.ai uses request-based pricing, so the cost is the same whether you send a 500-token summary or a 50,000-token detailed history. That makes it feasible to experiment with richer prompts that include full product descriptions, review snippets, and multi-turn session logs.

Here is a minimal example using the OpenAI SDK against Oxlo.ai:

import os
from openai import OpenAI

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key=os.environ["OXLO_API_KEY"]
)

def rank_candidates(user_history: list[dict], candidates: list[dict]) -> str:
    history_block = "\n".join(
        f"- {item['title']} ({item['category']}): {item['outcome']}"
        for item in user_history
    )
    candidate_block = "\n".join(
        f"{i+1}. {c['title']} - {c['description']}"
        for i, c in enumerate(candidates)
    )

    prompt = (
        "You are a personalized ranking assistant. "
        "Given the user's recent activity and a list of candidate items, "
        "return the candidate numbers sorted by relevance, most relevant first.\n\n"
        f"User history:\n{history_block}\n\n"
        f"Candidates:\n{candidate_block}\n\n"
        "Ranking:"
    )

    response = client.chat.completions.create(
        model="llama-3.3-70b",  # or deepseek-v3.2, qwen3-32b
        messages=[{"role": "user", "content": prompt}],
        temperature=0.2,
        max_tokens=512
    )
    return response.choices[0].message.content

# Example usage
ranking = rank_candidates(user_history, candidates[:20])

Because Oxlo.ai charges per request rather than per token, expanding the candidate_block from 10 to 50 items does not change the price of the call. You can iterate on prompt length without watching a meter run.

Using Embeddings for Semantic Retrieval

In some pipelines, you may want to replace or augment the collaborative filtering retrieval step with dense vector search over item descriptions. Oxlo.ai offers embedding endpoints for BGE-Large and E5-Large. You can index item descriptions into a vector store, then retrieve candidates that are semantically similar to the user’s stated preferences or recent positive interactions.

def embed_text(texts: list[str]) -> list[list[float]]:
    response = client.embeddings.create(
        model="bge-large",  # or e5-large
        input=texts
    )
    return [d.embedding for d in response.data]

item_embeddings = embed_text([c["description"] for c in catalog])
# Index with faiss, pgvector, or similar

Combining these embeddings with collaborative filtering gives you a true hybrid retriever. The LLM ranking layer then operates on candidates that are strong both behaviorally and semantically.

End-to-End Pipeline

Stitching the stages together looks like this:

1. Precompute ALS user and item factors nightly.
2. Precompute text embeddings for the catalog via Oxlo.ai embeddings.
3. At request time: a. Retrieve 100 candidates via ALS dot-product. b. Rerank the top 20 using an LLM prompt that includes the user’s last ten interactions and full item descriptions. c. Return the top 5.

The LLM step is the most expensive part of the pipeline on token-based platforms because it runs online and consumes long inputs. On Oxlo.ai, it is a single flat-cost request. For high-traffic applications, you can cache rankings for common user segments or use the streaming response feature to reduce time to first byte.

Cost Structure and Why It Matters

Token-based inference platforms bill you for every input and output token. In a recommender system, input tokens dominate: you are sending long user histories, product descriptions, and sometimes entire review threads. If your average ranking prompt is 8,000 tokens and you process a million rankings per day, token costs scale linearly with that volume and with the length of your context.

Oxlo.ai flips that model. You pay one flat fee per API request, regardless of prompt length. For long-context workloads like LLM-based ranking, that can yield substantial savings compared to token-based providers. You can see the exact plan details at https://oxlo.ai/pricing. The free tier includes 60 requests per day and access to 16+ models, which is enough to prototype a reranking pipeline before moving to a production plan.

Deployment Tips

Latency: Keep the candidate set small for the LLM stage. Twenty items is usually the sweet spot between quality and response time. Oxlo.ai serves popular models with no cold starts, so you will not pay a warmup penalty on the first request after idle time.

Caching: Store embeddings and ALS factors in Redis or an in-memory vector store. Cache LLM rankings for anonymous user segments or trending item bundles.

JSON mode: If you want structured output instead of free-text rankings, use the JSON mode feature to constrain the model to return a parseable array of item IDs and scores.

Monitoring: Because Oxlo.ai is OpenAI SDK compatible, you can reuse existing observability hooks. Track per-request latency and error rates exactly as you would with any OpenAI-compatible backend.

Conclusion

Building a recommender system that combines collaborative filtering with LLM reasoning gives you the best of both worlds: behavioral signal and semantic understanding. The main operational risk is runaway token costs as you enrich your prompts with more user context. Oxlo.ai removes that risk with flat per-request pricing, fully OpenAI SDK compatibility, and a broad catalog of open-source models. Whether you are prototyping on the free tier or running high-volume reranking in production, Oxlo.ai lets you maximize context without maximizing your bill.