Best Practices for Optimizing LLM Models for High Accuracy

#engineering #oxlo #ai

We are building a documentation QA agent that reads technical source text and answers questions with cited evidence. This is for support teams and developers who cannot afford hallucinated facts. We will optimize it step by step using prompt engineering, structured output, self-consistency, and a verification pass.

What you'll need

Python 3.10 or newer, the OpenAI SDK, and an Oxlo.ai API key from https://portal.oxlo.ai. Install the SDK with pip.

pip install openai

Step 1: Establish a baseline with a rigid system prompt

Accuracy starts with explicit instructions. I define a system prompt that forces the model to quote the source before concluding, which reduces hallucination. We will use Llama 3.3 70B on Oxlo.ai as our base model.

from openai import OpenAI

client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")

source_text = """The Oxlo.ai inference platform runs open-source and proprietary models with flat per-request pricing.
Unlike token-based providers, cost does not scale with prompt length.
Supported models include Llama 3.3 70B, Qwen 3 32B, and DeepSeek R1 671B MoE."""

question = "Does Oxlo.ai use token-based pricing?"

SYSTEM_PROMPT = """You are a rigorous technical QA agent. Answer questions using ONLY the provided context.

Rules:
1. Quote the exact sentences from the context that are relevant.
2. Reason step by step about what the quoted text implies.
3. Answer directly. If the context lacks the answer, say "Insufficient information."
4. Cite the quotes you used.
5. Do not use outside knowledge."""

response = client.chat.completions.create(
    model="llama-3.3-70b",
    messages=[
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user", "content": f"Context:\n{source_text}\n\nQuestion: {question}"},
    ],
)

print(response.choices[0].message.content)

Step 2: Lock output shape with JSON mode

Unstructured text invites parsing errors. JSON mode forces a machine-readable schema with reasoning, answer, citations, and confidence. I switch to Qwen 3 32B, which handles multilingual reasoning and agent instructions well.

import json

SYSTEM_PROMPT_JSON = SYSTEM_PROMPT + """
Return your response as a JSON object with exactly these keys:
- reasoning (string): your step-by-step thinking
- answer (string): the final concise answer
- citations (list of strings): the exact quotes you relied on
- confidence (integer 1-10): how certain you are"""

response = client.chat.completions.create(
    model="qwen-3-32b",
    messages=[
        {"role": "system", "content": SYSTEM_PROMPT_JSON},
        {"role": "user", "content": f"Context:\n{source_text}\n\nQuestion: {question}"},
    ],
    response_format={"type": "json_object"},
)

result = json.loads(response.choices[0].message.content)
print(json.dumps(result, indent=2))

Step 3: Add self-consistency sampling

A single sample can be unlucky. I run the agent three times and take the majority answer. Because Oxlo.ai charges per request rather than per token, this extra sampling stays predictable and is cheaper for long-context work than token-based alternatives. I use DeepSeek V3.2 for fast sampling.

from collections import Counter

def query_agent(context, question):
    resp = client.chat.completions.create(
        model="deepseek-v3.2",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT_JSON},
            {"role": "user", "content": f"Context:\n{context}\n\nQuestion: {question}"},
        ],
        response_format={"type": "json_object"},
    )
    return json.loads(resp.choices[0].message.content)

samples = [query_agent(source_text, question) for _ in range(3)]
answers = [s["answer"] for s in samples]
most_common = Counter(answers).most_common(1)[0][0]

best = next(s for s in samples if s["answer"] == most_common)
print(f"Consensus answer: {most_common}")
print(f"Confidence: {best['confidence']}")
print(f"Citations: {best['citations']}")

Step 4: Verify with a stronger reasoning model

Even consensus can be wrong. I add a second pass where a model specialized in reasoning critiques the proposed answer against the original text. Kimi K2.6 on Oxlo.ai excels at advanced reasoning and agentic coding, so I use it as the verifier.

VERIFIER_PROMPT = """You are a critical verification agent.
You will receive a context, a question, and a proposed answer with citations.
Check whether the proposed answer is fully supported by the context.
If you find unsupported claims or contradictions, provide a corrected answer.
Return JSON with keys:
- verdict (string: PASS, FAIL, or PARTIAL)
- corrected_answer (string)
- notes (string)"""

payload = f"""Context:
{source_text}

Question: {question}

Proposed answer: {best['answer']}
Citations: {best['citations']}
Confidence: {best['confidence']}"""

response = client.chat.completions.create(
    model="kimi-k2.6",
    messages=[
        {"role": "system", "content": VERIFIER_PROMPT},
        {"role": "user", "content": payload},
    ],
    response_format={"type": "json_object"},
)

verification = json.loads(response.choices[0].message.content)
print(json.dumps(verification, indent=2))

Run it

Here is the complete flow in one block. Call run_verified_qa with any context and question.

def run_verified_qa(context, question):
    # 1. Sample
    samples = [query_agent(context, question) for _ in range(3)]
    answers = [s["answer"] for s in samples]
    most_common = Counter(answers).most_common(1)[0][0]
    best = next(s for s in samples if s["answer"] == most_common)

    # 2. Verify
    payload = f"""Context:\n{context}\n\nQuestion: {question}\n\nProposed answer: {best['answer']}\nCitations: {best['citations']}"""
    resp = client.chat.completions.create(
        model="kimi-k2.6",
        messages=[
            {"role": "system", "content": VERIFIER_PROMPT},
            {"role": "user", "content": payload},
        ],
        response_format={"type": "json_object"},
    )
    v = json.loads(resp.choices[0].message.content)

    return {
        "draft": best,
        "verification": v,
        "final_answer": v.get("corrected_answer") or best["answer"]
    }

output = run_verified_qa(source_text, question)
print(json.dumps(output, indent=2))

Example output:

{
  "draft": {
    "reasoning": "The text explicitly states that Oxlo.ai uses 'flat per-request pricing' and that 'cost does not scale with prompt length', which directly contradicts token-based pricing.",
    "answer": "No, Oxlo.ai does not use token-based pricing.",
    "citations": [
      "Unlike token-based providers, cost does not scale with prompt length."
    ],
    "confidence": 10
  },
  "verification": {
    "verdict": "PASS",
    "corrected_answer": "",
    "notes": "The proposed answer is fully supported by the cited text. No corrections needed."
  },
  "final_answer": "No, Oxlo.ai does not use token-based pricing."
}

Wrap-up and next steps

This agent now reasons, cites, samples, and verifies. A concrete next step is to replace the hardcoded source_text with dynamic retrieval using Oxlo.ai's embedding models, such as BGE-Large, to fetch only the most relevant chunks before generation. Another step is to deploy the pipeline behind a FastAPI endpoint and stream the verification steps back to the client using Oxlo.ai's streaming response support.