Retrieval-augmented generation solved part of the hallucination problem, but unstructured text chunks still miss explicit relationships. A knowledge graph captures entities and their connections, giving an LLM structured context for precise question answering. Building this pipeline requires a capable reasoning model, a graph store, and an inference backend that does not penalize long prompts. Oxlo.ai offers a developer-first platform with request-based pricing and OpenAI SDK compatibility, which makes it a practical choice for agentic retrieval loops and long-context grounding workflows.
Architecture Overview
A typical graph-based QA system has four stages. First, an indexer parses source documents into entities and relations, then writes them into a graph database. Second, a retriever extracts entities from the user question and traverses the graph to collect a relevant subgraph. Third, a context assembler formats that subgraph into a prompt-friendly representation. Fourth, an LLM generates an answer conditioned on the structured context. Because subgraphs can grow large and iterative refinement often requires multiple LLM calls, inference costs can escalate quickly on token-based billing. Oxlo.ai flattens this cost with one price per request, so expanding context or adding verification steps does not inflate your bill.
Building the Knowledge Graph
You can store graphs in Neo4j, Amazon Neptune, or even an in-memory NetworkX graph for prototyping. The schema should distinguish entities from relations, and you should include textual descriptions on nodes so the LLM has something to read after traversal.
from typing import Dict, List, Tuple
class SimpleGraph:
def __init__(self):
self.nodes: Dict[str, Dict] = {}
self.edges: List[Tuple[str, str, str]] = []
def add_node(self, node_id: str, label: str, description: str):
self.nodes[node_id] = {"label": label, "description": description}
def add_edge(self, source: str, relation: str, target: str):
self.edges.append((source, relation, target))
def get_neighbors(self, node_id: str) -> List[Tuple[str, str, str]]:
results = []
for s, r, t in self.edges:
if s == node_id:
results.append((r, t, self.nodes.get(t, {}).get("description", "")))
elif t == node_id:
results.append((r, s, self.nodes.get(s, {}).get("description", "")))
return results
# Example: a tiny company knowledge base
graph = SimpleGraph()
graph.add_node("Alice", "Person", "Senior engineer on the platform team.")
graph.add_node("Oxlo.ai", "Company", "AI inference platform with flat per-request pricing.")
graph.add_node("Llama 3.3 70B", "Model", "General-purpose flagship model available on Oxlo.ai.")
graph.add_edge("Alice", "works_at", "Oxlo.ai")
graph.add_edge("Oxlo.ai", "offers", "Llama 3.3 70B")
Constructing the Retrieval Pipeline
Graph retrieval usually combines named entity recognition with one or more hops of neighborhood expansion. If your question is "Which engineer works at the company that offers Llama 3.3 70B?", a simple keyword matcher might find "Llama 3.3 70B", traverse backward to "Oxlo.ai", then forward to "Alice". For production systems, use an embedding index to map questions to candidate entities, then run breadth-first search to harvest the subgraph.
def retrieve_subgraph(question: str, graph: SimpleGraph, entry_nodes: List[str], depth: int = 2) -> str:
visited = set()
queue = [(n, 0) for n in entry_nodes]
chunks = []
while queue:
node_id, level = queue.pop(0)
if node_id in visited or level > depth:
continue
visited.add(node_id)
node = graph.nodes.get(node_id)
if node:
chunks.append(f"{node_id} ({node['label']}): {node['description']}")
for relation, neighbor_id, neighbor_desc in graph.get_neighbors(node_id):
if neighbor_id not in visited:
chunks.append(f" - {node_id} -[{relation}]-> {neighbor_id}: {neighbor_desc}")
queue.append((neighbor_id, level + 1))
return "\n".join(chunks)
# Pretend NER returned these entry points for our question
context = retrieve_subgraph(
"Which engineer works at the company that offers Llama 3.3 70B?",
graph,
entry_nodes=["Llama 3.3 70B"],
depth=2
)
print(context)
Integrating the LLM with Oxlo.ai
Once you have a structured context string, you need a model that follows instructions and can reason over relations. Oxlo.ai hosts several strong candidates, including DeepSeek R1 671B MoE for deep reasoning, Llama 3.3 70B for general-purpose performance, and Qwen 3 32B for agentic and multilingual workflows. All are accessible through the OpenAI SDK, so switching from another provider is a one-line URL change.
The key advantage for graph QA is Oxlo.ai's request-based pricing. Subgraph serialization can consume thousands of tokens, and agentic systems often rewrite queries or verify answers across multiple turns. On token-based providers, every additional token in the prompt raises cost. On Oxlo.ai, you pay per request, so you can pass large grounded contexts or run verification loops without watching input tokens accumulate. See https://oxlo.ai/pricing for current plan details.
import os
from openai import OpenAI
client = OpenAI(
base_url="https://api.oxlo.ai/v1",
api_key=os.environ.get("OXLO_API_KEY"),
)
SYSTEM_PROMPT = (
"You are a precise question-answering assistant. "
"Answer using only the facts in the provided knowledge graph context. "
"If the context does not contain the answer, say you do not know."
)
def answer_question(question: str, context: str, model: str = "llama-3.3-70b") -> str:
response = client.chat.completions.create(
model=model,
messages=[
{"role": "system", "content": SYSTEM_PROMPT},
{
"role": "user",
"content": f"Context:\n{context}\n\nQuestion: {question}",
},
],
temperature=0.1,
)
return response.choices[0].message.content
answer = answer_question(
"Which engineer works at the company that offers Llama 3.3 70B?",
context,
)
print(answer)
Putting It All Together
A minimal end-to-end script looks like the following. In production, replace the in-memory graph with Neo4j or RDFLib, add an embedding-based entity linker, and optionally use Oxlo.ai function calling to let the model request extra graph hops autonomously.
import os
from openai import OpenAI
class SimpleGraph:
def __init__(self):
self.nodes = {}
self.edges = []
def add_node(self, node_id, label, description):
self.nodes[node_id] = {"label": label, "description": description}
def add_edge(self, source, relation, target):
self.edges.append((source, relation, target))
def get_neighbors(self, node_id):
results = []
for s, r, t in self.edges:
if s == node_id:
results.append((r, t, self.nodes.get(t, {}).get("description", "")))
elif t == node_id:
results.append((r, s, self.nodes.get(s, {}).get("description", "")))
return results
def retrieve_subgraph(graph, entry_nodes, depth=2):
visited = set()
queue = [(n, 0) for n in entry_nodes]
chunks = []
while queue:
node_id, level = queue.pop(0)
if node_id in visited or level > depth:
continue
visited.add(node_id)
node = graph.nodes.get(node_id)
if node:
chunks.append(f"{node_id} ({node['label']}): {node['description']}")
for relation, neighbor_id, neighbor_desc in graph.get_neighbors(node_id):
if neighbor_id not in visited:
chunks.append(f" - {node_id} -[{relation}]-> {neighbor_id}: {neighbor_desc}")
queue.append((neighbor_id, level + 1))
return "\n".join(chunks)
# Build graph
graph = SimpleGraph()
graph.add_node("Alice", "Person", "Senior engineer on the platform team.")
graph.add_node("Oxlo.ai", "Company", "AI inference platform with flat per-request pricing.")
graph.add_node("Llama 3.3 70B", "Model", "General-purpose flagship model available on Oxlo.ai.")
graph.add_edge("Alice", "works_at", "Oxlo.ai")
graph.add_edge("Oxlo.ai", "offers", "Llama 3.3 70B")
# Retrieve context
context = retrieve_subgraph(graph, ["Llama 3.3 70B"], depth=2)
# Query Oxlo.ai
client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key=os.environ["OXLO_API_KEY"])
response = client.chat.completions.create(
model="llama-3.3-70b",
messages=[
{"role": "system", "content": "Answer using only the provided knowledge graph context."},
{"role": "user", "content": f"Context:\n{context}\n\nQuestion: Which engineer works at the company that offers Llama 3.3 70B?"},
],
temperature=0.1,
)
print(response.choices[0].message.content)
Deployment Considerations
Latency. Graph traversal plus LLM generation introduces two network hops. Oxlo.ai serves popular models with no cold starts, so the inference portion remains predictable even under load.
Context windows. Large enterprise graphs can produce subgraphs that exceed token limits. You can prune by relevance scoring, or you can switch to Oxlo.ai models that support very long contexts, such as DeepSeek V4 Flash with 1M context, for deep document graphs.
Updates. Knowledge graphs change over time. Version your graph snapshots and use Oxlo.ai JSON mode to enforce structured output when you need to diff or log model responses.
Conclusion
Combining a knowledge graph with an LLM gives you verifiable, structured answers instead of vague generations. The practical blocker is usually cost: long, structured prompts and iterative retrieval burn tokens fast. Oxlo.ai removes that friction with flat per-request pricing, OpenAI SDK compatibility, and a broad catalog of reasoning models. If you are building an agentic QA stack, start a free trial and point your client to https://api.oxlo.ai/v1.
Top comments (0)