LLM Text Summarization and Question Answering Guide

#learnai #oxlo #ai

Today we are building a document assistant that summarizes long articles and answers follow-up questions. I built this to process technical papers and internal wikis without setting up a separate vector database. Because Oxlo.ai uses request-based pricing, feeding the full text into the prompt on every call is affordable, so we can skip chunking and retrieval plumbing for many use cases.

What you'll need

Python 3.10 or newer
The OpenAI SDK: pip install openai
An Oxlo.ai API key from https://portal.oxlo.ai
A document to analyze. I will use a sample text about vector databases, but you can swap in your own.

Step 1: Initialize the client and load the document

Create a new file called doc_qa.py and add the client setup plus a source document. I am using a technical overview of vector databases as our target text.

from openai import OpenAI

client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")

DOCUMENT = """Vector databases have become a critical infrastructure component for modern AI applications. Unlike traditional relational databases that store data in rows and columns, vector databases are designed to store and query high-dimensional embeddings efficiently. When a user submits a query, the system converts the query into an embedding vector using a model such as BGE-Large or E5-Large. The database then performs an approximate nearest neighbor search to retrieve the most similar vectors. Popular algorithms for this include HNSW, IVF, and flat indexing. HNSW builds a multi-layer graph that enables logarithmic-time searches, but it requires significant memory overhead. IVF partitions the vector space into clusters and searches only the nearest clusters, offering a balance between speed and recall. Flat indexing is brute-force and exact, but becomes prohibitively slow at scale. Choosing the right index type depends on latency requirements, dataset size, and acceptable recall trade-offs. In production, developers must also consider sharding strategies, replication for high availability, and metadata filtering to combine vector similarity with structured queries. Recent advances include disk-based vector indexes that extend capacity beyond RAM limits, and sparse-dense hybrid retrieval that combines keyword and semantic matching. These systems power recommendation engines, retrieval-augmented generation pipelines, and semantic search interfaces."""

Step 2: Define the system prompt

The system prompt keeps the model focused on strict summarization and faithful question answering. I keep it short to minimize prompt engineering overhead.

SYSTEM_PROMPT = """You are a research assistant. Your job is to summarize documents and answer questions about them accurately.
Rules:
- Summarize the text into 3 to 5 bullet points covering the main ideas.
- When answering questions, use only the information in the provided text.
- If the answer is not in the text, say "The text does not mention this."""

Step 3: Generate a structured summary

We pass the full document in a single user message and ask for bullet points. On Oxlo.ai, this long input does not increase the cost because pricing is per request, not per token. That means we can send the entire article without chunking.

def summarize(text: str) -> str:
    user_message = f"Summarize the following text:\n\n{text}"
    response = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": user_message},
        ],
    )
    return response.choices[0].message.content

summary = summarize(DOCUMENT)
print("=== SUMMARY ===")
print(summary)

Step 4: Answer follow-up questions with context

For Q&A, I include the original document in every request alongside the question. This avoids retrieval errors and keeps the implementation stateless. Because Oxlo.ai charges a flat rate per request, repeating the full context is cost-effective compared to token-based providers.

def ask_question(text: str, question: str) -> str:
    user_message = f"Document:\n{text}\n\nQuestion: {question}"
    response = client.chat.completions.create(
        model="kimi-k2.6",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": user_message},
        ],
    )
    return response.choices[0].message.content

question = "What are the trade-offs between HNSW and IVF?"
answer = ask_question(DOCUMENT, question)
print("\n=== ANSWER ===")
print(answer)

Run it

Run the script from your terminal:

python doc_qa.py

Example output:

=== SUMMARY ===
- Vector databases store and query high-dimensional embeddings, unlike traditional relational databases.
- Common indexing algorithms include HNSW, IVF, and flat indexing, each with different trade-offs in speed, memory, and accuracy.
- Production deployments require attention to sharding, replication, and metadata filtering.
- Recent innovations include disk-based indexes and sparse-dense hybrid retrieval.
- These databases support recommendation engines, RAG pipelines, and semantic search.

=== ANSWER ===
HNSW builds a multi-layer graph that enables fast, logarithmic-time searches, but it requires significant memory overhead. IVF partitions the vector space into clusters and searches only the nearest clusters, which offers a balance between speed and recall. Flat indexing is exact but becomes prohibitively slow at scale.

Wrap-up and next steps

You now have a working document assistant that runs entirely on Oxlo.ai. Two concrete ways to extend it: wire the script into a Slack bot using Bolt so teammates can drop links and ask questions, or switch the output to JSON mode and feed structured summaries into a downstream analytics pipeline.

DEV Community