LLM Models for Question Answering: A Comprehensive Guide

#learnai #oxlo #ai

I built this agent for our internal support docs and refined it into the minimal version below. It answers questions strictly from provided source material and cites its sources, which helps support teams and researchers reduce hallucination when querying technical documentation.

What you'll need

Python 3.10 or newer, the OpenAI SDK (pip install openai), and an Oxlo.ai API key from https://portal.oxlo.ai. I also recommend setting your key as an environment variable named OXLO_API_KEY.

Step 1: Configure the Oxlo.ai client

First, initialize the OpenAI-compatible client pointing at Oxlo.ai and verify connectivity with a lightweight model call.

from openai import OpenAI
import os

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key=os.environ.get("OXLO_API_KEY", "YOUR_OXLO_API_KEY")
)

# Verify the endpoint is alive
response = client.chat.completions.create(
    model="llama-3.3-70b",
    messages=[{"role": "user", "content": "Say OK"}],
    max_tokens=5
)
print(response.choices[0].message.content)

Step 2: Prepare source material

I keep a short technical document in memory for this demo. The chunking function splits it into overlapping windows so we can inject only the most relevant pieces into the prompt.

DOCUMENT = """
Solar Panel Maintenance Guide v2.1
Installation: Panels must be tilted at 15 to 45 degrees for optimal yield.
Cleaning: Use deionized water and a soft brush. Do not use detergents.
Monitoring: Inverters report metrics every 15 minutes via Modbus TCP.
Troubleshooting: If yield drops by more than 20 percent, check for shading or inverter faults.
Safety: Disconnect DC isolators before performing any rooftop work.
"""

def chunk_text(text, chunk_size=40, overlap=10):
    words = text.split()
    chunks = []
    step = chunk_size - overlap
    for i in range(0, len(words), step):
        window = words[i:i + chunk_size]
        chunks.append(" ".join(window))
        if i + chunk_size >= len(words):
            break
    return chunks

chunks = chunk_text(DOCUMENT)
print(f"Created {len(chunks)} chunks")

Step 3: Write the system prompt

The system prompt constrains the model to the provided context and forces citations. This is the single most important guardrail against hallucination.

SYSTEM_PROMPT = """You are a precise technical support agent. Answer questions using only the provided context.
If the answer is not in the context, say 'I do not have enough information to answer that.'
Cite the relevant section at the end of your answer in this format: [Source: ].
Keep answers under 100 words unless the user asks for detail."""

Step 4: Build the QA function

I use a primitive keyword scorer to select the two most relevant chunks, then ask Llama 3.3 70B on Oxlo.ai to synthesize an answer. Because Oxlo.ai charges a flat rate per request instead of per token, I can pass larger context blocks during testing without worrying about ballooning costs. See https://oxlo.ai/pricing for plan details.

def retrieve_context(question, chunks):
    q_words = set(question.lower().split())
    scored = []
    for chunk in chunks:
        c_words = set(chunk.lower().split())
        overlap = len(q_words & c_words)
        scored.append((overlap, chunk))
    scored.sort(reverse=True)
    top = [c for s, c in scored[:2] if s > 0]
    if not top:
        top = chunks[:2]
    return "\n\n---\n\n".join(top)

def answer_question(question, chunks):
    context = retrieve_context(question, chunks)
    user_message = f"Context:\n{context}\n\nQuestion: {question}"
    
    response = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": user_message},
        ],
        temperature=0.1,
        max_tokens=300
    )
    return response.choices[0].message.content

Step 5: Add the interactive loop

Wire everything into a small CLI so you can query the document repeatedly.

if __name__ == "__main__":
    print("Document QA Agent ready. Type 'exit' to quit.")
    while True:
        question = input("\nQuestion: ").strip()
        if question.lower() in ("exit", "quit"):
            break
        if not question:
            continue
        print("\nAnswer:", answer_question(question, chunks))

Run it

Save the script as qa_agent.py, export your key, and run:

export OXLO_API_KEY="sk-..."
python qa_agent.py

Example session:

Document QA Agent ready. Type 'exit' to quit.

Question: What water should I use to clean panels?

Answer: Use deionized water and a soft brush. Do not use detergents. [Source: Cleaning]

Question: How often do inverters report?

Answer: Inverters report metrics every 15 minutes via Modbus TCP. [Source: Monitoring]

Question: Who invented the solar panel?

Answer: I do not have enough information to answer that.

Wrap-up and next steps

Swap in kimi-k2.6 or deepseek-v3.2 on Oxlo.ai if you need stronger reasoning for multi-hop questions, or switch to qwen-3-32b for multilingual documentation. For larger knowledge bases, replace the keyword retriever with Oxlo.ai's embedding endpoint using bge-large to rank chunks before injection.

DEV Community