Using LLM for Text Summarization and Question Answering

#learnai #oxlo #ai

We are building a document assistant that reads long articles or reports, produces a structured summary, and answers follow-up questions using the original text as ground truth. This is useful for research teams, support engineers, or anyone who needs to extract signal from dense material quickly.

What you'll need

Python 3.10 or newer
An Oxlo.ai API key from https://portal.oxlo.ai
The OpenAI SDK: pip install openai

Step 1: Configure the Oxlo.ai client

Replace YOUR_OXLO_API_KEY with the key from your Oxlo.ai dashboard. I use llama-3.3-70b as the default because it handles both summarization and reasoning in one call. If you are processing books or legal briefs, you can swap in kimi-k2.6 later for its 131K context window.

from openai import OpenAI

client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")

MODEL = "llama-3.3-70b"

Step 2: Define the system prompt

The system prompt needs to be strict. I tell the model to ground every answer in the provided text, avoid speculation, and format summaries as bullet points. I store it as a constant so I can tweak it without touching business logic.

SYSTEM_PROMPT = """You are a precise document assistant. Your job has two parts:

1. Summarization: When the user provides a document under the tag <document>, output a structured summary with three sections: Key Points, Named Entities, and One-Sentence Takeaway.
2. Question Answering: When the user asks a question, answer using only the information inside the most recent <document> tags. If the answer is not in the text, say "The document does not specify."

Rules:
- Use bullet points for lists.
- Do not invent facts.
- Keep answers under 150 words unless the user asks for detail."""

Step 3: Summarize long documents

I wrap the raw text in XML-like tags so the model knows what is source material versus instruction. This reduces prompt injection and keeps the context boundary clean. Because Oxlo.ai uses request-based pricing, sending a 10K word article in a single call costs the same as a one-liner, which makes long-context summarization practical without token math.

def summarize_document(text: str) -> str:
    user_message = f"Please summarize the following document:\n\n<document>\n{text}\n</document>"
    
    response = client.chat.completions.create(
        model=MODEL,
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": user_message},
        ],
        temperature=0.3,
        max_tokens=1024,
    )
    return response.choices[0].message.content

Step 4: Answer questions from source

For QA, I append the question after the document block so the model has the full source in the same context window. This avoids retrieval errors and keeps the implementation stateless. If your documents routinely exceed the context limit, chunk them and run map-reduce, but for most reports a single call works fine.

def ask_question(text: str, question: str) -> str:
    user_message = (
        f"<document>\n{text}\n</document>\n\n"
        f"Question: {question}"
    )
    
    response = client.chat.completions.create(
        model=MODEL,
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": user_message},
        ],
        temperature=0.1,
        max_tokens=512,
    )
    return response.choices[0].message.content

Step 5: Wire the interactive loop

I tie the two functions together in a small CLI. It loads a sample document, prints the summary, then enters a loop where the user can ask questions. In production you would swap stdin for an API endpoint, but this version is enough to validate the behavior.

SAMPLE_DOC = """
Artificial intelligence adoption in enterprise software has accelerated since 2023.
Key drivers include API-based access to large language models, reduced infrastructure costs,
and improved retrieval pipelines. Security remains the primary blocker, with 60 percent of
CIOs citing data residency as a top concern. Vendors who offer self-hosted or VPC deployment
options are winning evaluations in regulated industries. The report predicts that by 2026,
over 50 percent of new enterprise applications will embed LLM-powered features natively.
"""

if __name__ == "__main__":
    print("Generating summary...\n")
    summary = summarize_document(SAMPLE_DOC)
    print(summary)
    print("\n---\n")

    while True:
        try:
            q = input("Ask a question (or type 'quit'): ").strip()
            if q.lower() in ("quit", "exit"):
                break
            answer = ask_question(SAMPLE_DOC, q)
            print(f"\nAnswer: {answer}\n")
        except KeyboardInterrupt:
            break

Run it

Save the script as doc_agent.py, set your YOUR_OXLO_API_KEY, then run python doc_agent.py. You should see output similar to this:

Generating summary...

Key Points:
- Enterprise AI adoption has accelerated since 2023 due to API access, lower costs, and better retrieval.
- Security and data residency are the main blockers for CIOs.
- Self-hosted or VPC options are winning in regulated industries.
- By 2026, over 50 percent of new enterprise apps will embed LLM features natively.

Named Entities:
- CIOs, regulated industries, enterprise software vendors.

One-Sentence Takeaway:
- Enterprise AI is growing rapidly, but security and deployment flexibility are deciding factors in procurement.

---

Ask a question (or type 'quit'): What do CIOs care about most?
Answer: The document states that 60 percent of CIOs cite data residency as a top concern.

Ask a question (or type 'quit'): Who wrote the report?
Answer: The document does not specify.

Wrap-up and next steps

Swap in kimi-k2.6 if you need to process full PDFs without chunking, since its 131K context window handles most books in one shot. If you want to expose this as a service, wrap the functions in FastAPI and stream responses using Oxlo.ai's streaming support. You can view request-based pricing details at https://oxlo.ai/pricing.

DEV Community