🚀 Technical Briefing: This tutorial is part of our deep-dive series on Agentic Workflows at Gate of AI. For the full technical breakdown, interactive code sandbox, and the native Arabic translation, visit the original article here.
Build a local, dependency-free sentence-window retrieval prototype, understand why precise retrieval needs surrounding context, and evaluate the evidence before connecting the pattern to a production RAG stack.
What this tutorial covers
Retrieval-augmented generation, usually shortened to RAG, gives an answer system external text to consult at query time. A common implementation choice is to split every document into fixed-size chunks and retrieve the chunks most related to a question. That approach is useful, but it creates a persistent design trade-off. Small chunks can make retrieval precise while removing definitions, conditions, and exceptions. Large chunks can restore context while adding irrelevant material to the prompt.
Sentence-window retrieval addresses that trade-off by separating the unit used for retrieval from the unit used for interpretation. The system indexes individual sentences. When a sentence is selected, the system expands it into a local window containing nearby sentences from the same document. The retrieval signal remains precise, while the reader receives a fuller passage.
This tutorial intentionally uses only the Python standard library. The verified context does not establish current APIs, package versions, model availability, or persistence behavior for a particular RAG framework or model provider. A local prototype is therefore the most accurate way to demonstrate the technique without presenting unverified architecture as fact. Once the behavior is understood and tested, map the same concepts to the components that your organization has independently verified.
Why local context matters in RAG
A sentence often contains the words that best match a user question but not the complete meaning. Consider a policy passage with a rule, an exception, and a deadline. A query may match the sentence containing the deadline, while the preceding sentence says the policy applies only to a particular role. Returning the deadline alone can create a misleading answer.
The verified research context identifies a related problem in conventional RAG: retrieving too much information can create token-limit pressure and the “lost in the middle” problem, where relevant details become less useful among excessive context. The same research proposes retrieving chunks at multiple abstraction levels, including multi-sentence, paragraph, section, and document levels. In its Glycoscience-paper evaluation, that approach improved AI-evaluated question-answer correctness by 25.739% compared with a traditional single-level approach. This is a research result for that evaluation, not a promise that every corpus or sentence-window configuration will improve by the same amount.
A sentence window is one practical multi-sentence context pattern. It is especially appropriate when facts and their qualifications are usually located near each other. It is less suitable when the evidence required to answer a question is dispersed across distant sections or multiple documents. In those cases, a system may need broader retrieval, additional abstraction levels, or a document structure designed for the task.
Prerequisites
- Python 3.10 or later.
- A terminal capable of running Python commands.
- A small set of trusted UTF-8 plain-text or Markdown documents.
- Familiarity with basic command-line navigation and Python files.
This prototype does not call a model API. It retrieves evidence and prints the selected context windows. That boundary is deliberate: it lets you inspect whether retrieval has selected adequate evidence before introducing answer generation.
Step 1: Create a small corpus with rules and exceptions
Create a project directory and two short Markdown documents. The sample corpus is fictional. Its purpose is to make it easy to see why a sentence match alone may not carry enough context.
mkdir sentence-window-rag
cd sentence-window-rag
mkdir data
cat > data/travel_policy.md <<'EOF'
# Travel Policy
Employees must use the approved travel portal when inventory is available. Economy class is required for flights shorter than six hours. Premium economy may be booked for flights of six hours or longer.
Business class requires written approval from a vice president before booking. A manager approval is not sufficient. The approval email must be attached to the expense report.
EOF
cat > data/security_policy.md <<'EOF'
# Security Policy
Privileged production access requires multi-factor authentication and an approved access request. Shared user accounts are prohibited. Temporary production access expires automatically after eight hours unless an incident commander extends it during an active incident.
Employees must report suspected security incidents immediately through the incident portal. If the portal is unavailable, employees must contact the on-call security engineer.
EOF
Keep documents that you index within the authorization boundary of the intended users. This local example has no authentication, filtering, or remote service. Do not treat it as a ready-made system for confidential documents.
Step 2: Build a sentence index and local context windows
Create sentence_window_rag.py. The script reads Markdown and text files, separates text into simple sentence-like units, calculates a transparent lexical relevance score, and expands every result into neighboring sentences from the same source file. It is a learning implementation, not a linguistic sentence parser or a semantic-vector retrieval engine.
from __future__ import annotations
import argparse
import math
import re
from collections import Counter
from dataclasses import dataclass
from pathlib import Path
TOKEN_PATTERN = re.compile(r"[a-z0-9]+")
SENTENCE_PATTERN = re.compile(r"(?<=[.!?])\s+")
@dataclass(frozen=True)
class SentenceRecord:
source_file: str
position: int
text: str
def tokenize(text: str) -> list[str]:
return TOKEN_PATTERN.findall(text.lower())
def split_sentences(text: str) -> list[str]:
cleaned = re.sub(r"^#+\s+.*$", "", text, flags=re.MULTILINE)
cleaned = re.sub(r"\s+", " ", cleaned).strip()
if not cleaned:
return []
return [part.strip() for part in SENTENCE_PATTERN.split(cleaned) if part.strip()]
def load_records(data_dir: Path) -> list[SentenceRecord]:
records: list[SentenceRecord] = []
for path in sorted(data_dir.rglob("*")):
if not path.is_file() or path.suffix.lower() not in {".md", ".txt"}:
continue
text = path.read_text(encoding="utf-8")
for position, sentence in enumerate(split_sentences(text)):
records.append(
SentenceRecord(
source_file=path.name,
position=position,
text=sentence,
)
)
if not records:
raise ValueError("No non-empty .md or .txt sentences were found in the data directory.")
return records
def inverse_document_frequency(records: list[SentenceRecord]) -> dict[str, float]:
document_frequency: Counter[str] = Counter()
for record in records:
document_frequency.update(set(tokenize(record.text)))
total = len(records)
return {
token: math.log((total + 1) / (count + 1)) + 1.0
for token, count in document_frequency.items()
}
def score(question: str, sentence: str, idf: dict[str, float]) -> float:
question_terms = Counter(tokenize(question))
sentence_terms = Counter(tokenize(sentence))
if not question_terms or not sentence_terms:
return 0.0
numerator = sum(
question_terms[token] * sentence_terms[token] * (idf.get(token, 0.0) ** 2)
for token in question_terms
)
question_norm = math.sqrt(
sum((count * idf.get(token, 0.0)) ** 2 for token, count in question_terms.items())
)
sentence_norm = math.sqrt(
sum((count * idf.get(token, 0.0)) ** 2 for token, count in sentence_terms.items())
)
if question_norm == 0.0 or sentence_norm == 0.0:
return 0.0
return numerator / (question_norm * sentence_norm)
def context_window(records: list[SentenceRecord], record: SentenceRecord, radius: int) -> str:
same_file = [item for item in records if item.source_file == record.source_file]
start = max(0, record.position - radius)
end = min(len(same_file), record.position + radius + 1)
return " ".join(item.text for item in same_file[start:end])
def search(records: list[SentenceRecord], question: str, top_k: int, radius: int):
idf = inverse_document_frequency(records)
ranked = sorted(
((score(question, record.text, idf), record) for record in records),
key=lambda item: item[0],
reverse=True,
)
return [
(relevance, record, context_window(records, record, radius))
for relevance, record in ranked[:top_k]
if relevance > 0.0
]
def main() -> None:
parser = argparse.ArgumentParser(description="Inspect sentence-window retrieval.")
parser.add_argument("question", help="Question to search for")
parser.add_argument("--data-dir", default="data")
parser.add_argument("--top-k", type=int, default=3)
parser.add_argument("--window", type=int, default=1)
args = parser.parse_args()
if args.top_k < 1 or args.window < 0:
raise SystemExit("--top-k must be at least 1 and --window must be zero or greater.")
records = load_records(Path(args.data_dir))
results = search(records, args.question, args.top_k, args.window)
if not results:
print("No lexical overlap was found. This prototype should abstain rather than answer.")
return
for number, (relevance, record, window) in enumerate(results, start=1):
print(f"Result {number}")
print(f"Source: {record.source_file}")
print(f"Sentence position: {record.position}")
print(f"Lexical score: {relevance:.4f}")
print(f"Retrieved sentence: {record.text}")
print(f"Context window: {window}\n")
if __name__ == "__main__":
main()
The retrieved sentence is the narrow evidence unit. The context window is the expanded evidence unit. The --window value is a radius: a value of 1 includes the selected sentence plus up to one preceding and one following sentence. Document boundaries limit the window automatically.
Step 3: Run evidence-first queries
Run the following commands. Start with a one-sentence radius, then compare the output with a radius of zero. The difference demonstrates why a sentence may be a strong retrieval match but a weak standalone citation.
python sentence_window_rag.py "Who can approve business class travel?" --window 1
python sentence_window_rag.py "Who can approve business class travel?" --window 0
python sentence_window_rag.py "What are the requirements for temporary production access?" --window 1
python sentence_window_rag.py "What is the parental leave policy?" --window 1
For business-class travel, inspect whether the window contains both the vice-president requirement and the statement that manager approval is insufficient. For temporary production access, inspect whether the selected window contains the authentication requirement, the approved access request, the eight-hour expiry, and the incident-commander exception. The exact ranking is not the lesson; this prototype uses lexical scoring rather than semantic embeddings. The lesson is that the answerer should see the surrounding conditions before producing an answer.
The parental-leave query is an abstention test. The sample corpus has no relevant source. A trustworthy next stage should not convert unrelated travel or security passages into an invented policy. Retaining the retrieved evidence in the result makes this failure visible to a reviewer.
Step 4: Evaluate the retrieval design before adding generation
Do not judge a RAG design only by whether it produces fluent prose. Evaluate it in layers. First, ask whether the correct source passage appears among the retrieved results. Second, ask whether the selected window includes the qualifications necessary to interpret that passage. Third, once an answer component is added, ask whether each material statement in the answer is supported by the selected evidence. Finally, test whether the system abstains when the corpus does not contain an answer.
Create a compact evaluation file such as evaluation.jsonl. Each record can include a question, expected source file, required concepts, and whether abstention is expected. For example, the travel question should require the concepts “written approval,” “vice president,” and “manager approval is not sufficient.” The access question should require “multi-factor authentication,” “approved access request,” “eight hours,” and the active-incident exception.
Run the same evaluation set when you adjust sentence splitting, window size, ranking method, document formatting, or the downstream answer prompt. This turns tuning into a comparison process rather than an anecdotal exercise. A larger window is not automatically better: it can add useful conditions, but it can also add unrelated language that distracts an answer system. Likewise, retrieving more sentences can improve recall while increasing context volume.
Step 5: Connect the pattern to a production stack carefully
The local script is not a production service. It does not provide semantic retrieval, access control, document-version management, API authentication, concurrency controls, or answer generation. Those are separate design decisions. When moving to a verified framework and provider stack, preserve the core sequence: parse trusted documents into sentence-level retrieval units; store a local window as associated context; retrieve narrow evidence; replace or supplement the retrieval unit with its window; present citations alongside any generated answer; and measure retrieval and answer support independently.
Use the smallest context that reliably retains key conditions. If questions commonly require information distributed across paragraphs, sections, or documents, evaluate multiple retrieval abstraction levels rather than assuming a fixed sentence window will solve every case. This is consistent with the verified research context: information needs can occur at more than one level of abstraction, while excessive retrieved text can harm usefulness.
For sensitive or regulated material, apply authorization before text is selected for an answer workflow. Maintain an evaluation corpus that reflects the documents and users your system actually serves. Avoid presenting research benchmarks as deployment guarantees, and avoid relying on a prompt alone to compensate for missing evidence or inappropriate retrieval.
Key takeaways
- Sentence-window RAG retrieves a precise sentence and expands it with nearby context from the same document.
- The pattern helps preserve nearby rules, exceptions, thresholds, and deadlines that a single sentence may omit.
- More context is not always better; excessive context can contribute to token pressure and lost-in-the-middle behavior.
- Evaluate source selection, context completeness, answer support, and abstention separately.
- The reported 25.739% correctness improvement belongs to a specific multi-abstraction research evaluation on Glycoscience papers and should not be generalized as a universal result.
Sources
- Multiple Abstraction Level Retrieve Augment Generation, arXiv:2501.16952v1. Verified context for multi-level retrieval, token-limit and lost-in-the-middle considerations, and the 25.739% reported Glycoscience evaluation result.
- Experience Retrieval-Augmentation with Electronic Health Records Enables Accurate Discharge QA, arXiv:2503.17933v1. Verified context showing that retrieval design can use task-relevant, case-grounded information in a clinical question-answering research setting.
Top comments (0)