Building a Question Answering System with LLM and Knowledge Graph for Enterprise Search

#engineering #oxlo #ai

We are building a retrieval-augmented question answering system that combines a Neo4j knowledge graph with an LLM to answer complex enterprise search queries. This helps teams navigate interconnected documents, people, and projects without writing Cypher or SQL. I will walk through the entire pipeline from graph construction to natural language answers.

What you'll need

Python 3.10+, a running Neo4j instance (local or AuraDB), and the neo4j and openai Python packages. You also need an Oxlo.ai API key from https://portal.oxlo.ai. Oxlo.ai is a good fit here because we will make multiple LLM calls with long system prompts, and its flat per-request pricing keeps costs predictable regardless of prompt size. See https://oxlo.ai/pricing for details.

pip install openai neo4j

Step 1: Seed the graph with enterprise data

First, connect to Neo4j and insert sample nodes for employees, projects, and documents. This schema is what the LLM will query against.

from neo4j import GraphDatabase

URI = "bolt://localhost:7687"
AUTH = ("neo4j", "password")

def seed_graph():
    driver = GraphDatabase.driver(URI, auth=AUTH)
    with driver.session() as session:
        session.run("MATCH (n) DETACH DELETE n")
        session.run("""
            CREATE (alice:Employee {name: 'Alice Chen', role: 'Senior Engineer', department: 'Platform'})
            CREATE (bob:Employee {name: 'Bob Smith', role: 'Security Lead', department: 'Security'})
            CREATE (carol:Employee {name: 'Carol Jones', role: 'Product Manager', department: 'Product'})
            CREATE (p:Project {name: 'Phoenix', status: 'active'})
            CREATE (d1:Document {title: 'Compliance 2024 Audit', summary: 'Annual security compliance findings.'})
            CREATE (d2:Document {title: 'Platform Migration Guide', summary: 'Steps to migrate services.'})
            CREATE (alice)-[:WORKS_ON]->(p)
            CREATE (bob)-[:WORKS_ON]->(p)
            CREATE (carol)-[:WORKS_ON]->(p)
            CREATE (alice)-[:AUTHORED]->(d2)
            CREATE (bob)-[:AUTHORED]->(d1)
            CREATE (p)-[:REFERENCES]->(d1)
        """)
    driver.close()
    print("Graph seeded.")

if __name__ == "__main__":
    seed_graph()

Step 2: Generate Cypher queries with an LLM

We will use Oxlo.ai to translate natural language into Cypher. Because the system prompt includes the full schema description, prompt length grows quickly. Oxlo.ai's flat per-request pricing keeps generation costs predictable no matter how detailed the schema gets, which makes it a strong fit for this pipeline.

from openai import OpenAI

client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")

CYPHER_SYSTEM_PROMPT = """
You are a Cypher expert for an enterprise knowledge graph.
The graph schema is:
- (:Employee {name, role, department})
- (:Project {name, status})
- (:Document {title, summary})
- (:Employee)-[:WORKS_ON]->(:Project)
- (:Employee)-[:AUTHORED]->(:Document)
- (:Project)-[:REFERENCES]->(:Document)

Rules:
- Return only the Cypher query, no markdown, no explanation.
- Use only READ operations. Never use CREATE, MERGE, DELETE, SET, or REMOVE.
- Always limit results to 20 rows unless the user asks otherwise.
- If the question cannot be answered with this schema, return "UNSUPPORTED".
"""

def generate_cypher(question: str) -> str:
    response = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=[
            {"role": "system", "content": CYPHER_SYSTEM_PROMPT},
            {"role": "user", "content": question},
        ],
        temperature=0.1,
    )
    query = response.choices[0].message.content.strip()
    query = query.replace("

```cypher", "").replace("```

", "").strip()
    return query

Step 3: Execute queries and safeguard the database

Before running generated Cypher, we strip markdown fences and block any write keywords. Then we execute against Neo4j and return clean dictionaries.

FORBIDDEN_KEYWORDS = ["create", "merge", "delete", "set", "remove", "drop"]

def is_read_only(query: str) -> bool:
    return not any(kw in query.lower() for kw in FORBIDDEN_KEYWORDS)

def run_cypher(query: str):
    if not is_read_only(query):
        raise ValueError("Query blocked: write operations are not allowed.")
    
    driver = GraphDatabase.driver(URI, auth=AUTH)
    with driver.session() as session:
        result = session.run(query)
        records = [dict(r) for r in result]
    driver.close()
    return records

Step 4: Synthesize answers from graph results

Raw graph records are not user-friendly. We pass the JSON results back to Oxlo.ai along with the original question and ask for a concise, cited answer.

ANSWER_SYSTEM_PROMPT = """
You are an enterprise search assistant. Answer the user's question using only the provided graph results.
Be concise, accurate, and cite names or titles explicitly. If the results are empty, say you could not find the information.
"""

def synthesize(question: str, records: list) -> str:
    context = f"Question: {question}\nGraph results: {records}"
    response = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=[
            {"role": "system", "content": ANSWER_SYSTEM_PROMPT},
            {"role": "user", "content": context},
        ],
        temperature=0.2,
    )
    return response.choices[0].message.content.strip()

Step 5: Wire everything into a single QA agent

This function orchestrates the full loop: generate Cypher, validate and run it, then synthesize the final response.

def enterprise_qa(question: str) -> dict:
    cypher = generate_cypher(question)
    if cypher == "UNSUPPORTED":
        return {
            "question": question,
            "cypher": None,
            "answer": "I cannot answer that with the current graph schema."
        }
    
    records = run_cypher(cypher)
    answer = synthesize(question, records)
    return {
        "question": question,
        "cypher": cypher,
        "records": records,
        "answer": answer
    }

Run it

Call the agent with a natural language question and inspect the generated Cypher and final answer.

if __name__ == "__main__":
    seed_graph()
    q = "Which employees working on Project Phoenix authored a compliance document?"
    result = enterprise_qa(q)
    print("Generated Cypher:")
    print(result["cypher"])
    print("\nAnswer:")
    print(result["answer"])

Example output:

Generated Cypher:
MATCH (e:Employee)-[:WORKS_ON]->(p:Project {name: 'Phoenix'}), (e)-[:AUTHORED]->(d:Document)
WHERE d.title CONTAINS 'Compliance' RETURN e.name, d.title LIMIT 20

Answer:
Bob Smith, who works on Project Phoenix, authored the document "Compliance 2024 Audit".

Next steps

Add vector similarity search over document content by generating embeddings with Oxlo.ai's embedding models, then hybridize graph traversal with semantic retrieval for richer answers. Alternatively, expose this pipeline through a FastAPI endpoint so your internal tools can query it directly.

DEV Community