Building a Recommender System with LLM and Low Latency

Recommender systems are moving past collaborative filtering and two-tower embeddings toward LLM-driven ranking and generation. The upside is richer context understanding, but the production constraint is always latency. When a user refreshes a feed, every millisecond matters. This article outlines a practical two-stage architecture that keeps latency low, and explains why your inference provider’s pricing model directly impacts how much context you can afford to send.

Architecture: Retrieve, Then Rerank

A single monolithic LLM call that scans an entire catalog is too slow and too expensive, even under flat pricing. The standard pattern remains a two-stage pipeline.

1. Retrieval: Use an embedding model to encode user history and candidate items, then pull the top-K via vector similarity.
2. Reranking: Feed the retrieved subset into an LLM with a structured prompt. The model returns a ranked list, or generates a short explanation.

This bounds the LLM prompt size to a small set of candidates, which keeps generation latency predictable. The heavy lifting happens in the embedding step, which is trivially parallelizable and cacheable.

Latency Levers in Production

Beyond basic caching, three decisions determine your p99.

• Model size versus quality. A 70B parameter model produces better reasoning than a 7B model, but TTFT (time to first token) is higher. For ranking tasks, a strong 32B or 70B model is often the sweet spot. Oxlo.ai hosts Llama 3.3 70B and Qwen 3 32B with no cold starts, so you do not pay a warmup penalty on the first request after idle time.
• Structured output. Forcing the model to emit JSON via response_format={"type": "json_object"} removes the need for client-side regex parsing and reduces follow-up turns.
• Prompt length. Recommender prompts swell quickly when you include user session history, product attributes, and review summaries. On token-based providers, long prompts inflate cost linearly. On Oxlo.ai, request-based pricing means the cost stays flat regardless of how much context you pack into a single request.

A Minimal Implementation

The following Python snippet uses the OpenAI SDK pointed at Oxlo.ai. It embeds user history and candidate items using an embedding model, retrieves the top candidates by cosine similarity, then asks a chat model to rerank them.

import os
import json
import numpy as np
from openai import OpenAI

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key=os.environ["OXLO_API_KEY"]
)

def embed_texts(texts):
    # Oxlo.ai offers BGE-Large and E5-Large for embeddings
    resp = client.embeddings.create(
        model="BGE-Large",  # use the exact model ID from the Oxlo.ai catalog
        input=texts
    )
    return [d.embedding for d in resp.data]

def retrieve(user_history, items, k=10):
    texts = [user_history] + [item["description"] for item in items]
    vectors = embed_texts(texts)
    user_vec = np.array(vectors[0])
    item_vecs = np.array(vectors[1:])
    # cosine similarity via dot product over L2-normalized vectors
    scores = item_vecs @ user_vec
    top_k = np.argpartition(scores, -k)[-k:]
    return [items[i] for i in top_k[np.argsort(scores[top_k])[::-1]]]

def rerank(user_history, candidates):
    candidate_block = "\n".join(
        f"{i+1}. ID:{c['id']} | {c['description']}"
        for i, c in enumerate(candidates)
    )
    messages = [
        {"role": "system", "content": "You rank items by relevance. Return strict JSON: {\"ranked_ids\": [...]}"},
        {"role": "user", "content": f"User history: {user_history}\nItems:\n{candidate_block}\nReturn only JSON."}
    ]
    resp = client.chat.completions.create(
        model="Llama-3.3-70B",  # exact model ID from the Oxlo.ai catalog
        messages=messages,
        response_format={"type": "json_object"},
        temperature=0.1,
        max_tokens=512
    )
    return json.loads(resp.choices[0].message.content)

# Example usage
catalog = [
    {"id": "sku-1", "description": "Waterproof hiking boots, durable leather"},
    {"id": "sku-2", "description": "Trail running shoes, lightweight mesh"},
    # ... full catalog
]

history = "Bought camping tent last week, looking for footwear."
top_candidates = retrieve(history, catalog, k=5)
result = rerank(history, top_candidates)
print(result["ranked_ids"])

Note the response_format flag. Oxlo.ai supports JSON mode across its chat models, so you can enforce structured output without additional parsing logic.

Why Pricing Model Determines Context Budget

Recommender quality improves when the model sees more signals: recent clicks, dwell time, product specs, and even raw review text. On token-based providers such as Together AI, Fireworks AI, OpenRouter, Replicate, or Anyscale, stuffing that context into every request multiplies your bill. Teams end up trimming user history or stripping metadata to save money, which directly hurts recommendation quality.

Oxlo.ai uses flat per-request pricing. One API call costs the same whether you send a 200-token prompt or a 20,000-token prompt. For long-context recommender workloads, this can be 10-100x cheaper than token-based billing. That pricing structure changes engineering decisions. You can afford to send the full user session, detailed item descriptions, and even multi-turn conversational context in a single request without cost surprises.

If you truly need extreme context, DeepSeek V4 Flash on Oxlo.ai supports 1M tokens and efficient MoE inference, useful for ranking over large retrieved sets in one shot. For most production pipelines, Llama 3.3 70B or Kimi K2.6 provide the best balance of reasoning quality and throughput.

Benchmarking for Production

Latency is more than TTFT. Measure end-to-end wall time from request serialization to parsed JSON. Run load tests that include cache misses and embedding fan-out. Because Oxlo.ai has no cold starts on popular models, your p99 should stay flat even after idle periods, which is critical for recommender systems that see bursty traffic.

Start with the free tier to baseline latency against your SLA. Oxlo.ai offers 60 requests per day on 16-plus models, including a 7-day full-access trial, which is enough to profile a two-stage pipeline on real data. When you scale, the Pro and Premium plans offer 1,000 and 5,000 requests per day respectively, with priority queueing on Premium. See https://oxlo.ai/pricing for current plan details.

Conclusion

LLM-powered recommenders are production-ready when you bound the problem: embed at scale, retrieve a small candidate set, and rerank with structured generation. The remaining variable is inference economics. Flat per-request pricing on Oxlo.ai removes the penalty for rich context, letting you ship better recommendations without trimming user history or item metadata to fit a token budget. If you are migrating from a token-based provider, the OpenAI SDK compatibility means the code change is a single base_url swap. Point your client to https://api.oxlo.ai/v1 and test the difference on your own latency metrics.