We are building a long-context document analysis agent that reads an entire article and answers complex questions requiring the model to attend to multiple sections. This demonstrates how attention mechanisms in modern LLMs handle long-range dependencies in NLP tasks. Because Oxlo.ai charges a flat rate per request, running this on full documents is cost-predictable regardless of input length.
What you'll need
- Python 3.10 or newer
- The OpenAI SDK:
pip install openai - An Oxlo.ai API key from https://portal.oxlo.ai
Step 1: Set up the Oxlo.ai client
Initialize the OpenAI-compatible client pointing at Oxlo.ai. I use llama-3.3-70b as the backbone because it handles long-context reasoning reliably.
from openai import OpenAI
client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")
MODEL = "llama-3.3-70b"
Step 2: Write the system prompt
The system prompt instructs the model to cite specific sections and reason across the full context. This forces the underlying attention mechanism to locate relevant spans rather than hallucinate.
SYSTEM_PROMPT = """You are a precise document analyst. Your job is to read the entire document provided by the user and answer questions based only on its contents.
Rules:
1. Cite the specific paragraph or section that supports each part of your answer.
2. If the answer requires combining information from multiple distant sections, explain the connection explicitly.
3. Do not use outside knowledge. If the document does not contain the answer, say so clearly.
4. Keep your answer concise but thorough."""
Step 3: Load a long document
I need a document long enough that simple keyword search fails but attention-based reasoning succeeds. I embed a lengthy overview of attention mechanisms in NLP. In production, this would be a PDF or log file.
DOCUMENT = """
Attention Mechanisms in Natural Language Processing: A Survey
1. Introduction
Sequence-to-sequence models revolutionized machine translation by encoding an input sentence into a fixed-length vector and decoding it into another language. However, the fixed-length bottleneck made it difficult to handle long sentences. Bahdanau et al. proposed the first neural attention mechanism for sequence modeling, allowing the decoder to focus on different parts of the source sentence at each step.
2. From RNN Attention to Transformers
Early attention mechanisms were tied to recurrent networks. The encoder hidden states served as a memory bank, and the decoder computed a weighted sum over these states. The weights, determined by a compatibility function, indicated where to attend. This approach improved translation quality but remained sequential and slow to train.
Vaswani et al. removed recurrence entirely with the Transformer architecture. Self-attention computes relationships between all pairs of tokens in parallel. The key insight is that each token can directly attend to any other token, capturing long-range dependencies in a constant number of layers. This parallelism enabled training on unprecedented corpus sizes.
3. Multi-Head and Scaled Dot-Product Attention
The Transformer uses scaled dot-product attention. Given queries Q, keys K, and values V, the output is softmax(QK^T / sqrt(d_k))V. Multi-head attention runs this computation h times with different learned projections, allowing the model to attend to information from different representation subspaces.
4. Long-Context Challenges
As documents grow, the memory and compute for full self-attention scale quadratically with sequence length. Sparse attention patterns, sliding windows, and low-rank approximations have been proposed to reduce this burden. Recent models have pushed effective context windows from 2,048 tokens to over one million tokens through careful architectural changes and positional extrapolation.
5. Applications in NLP
Attention underpins modern NLP. In sentiment analysis, attention weights highlight words that drive polarity. In question answering, models attend to passages that contain the answer span. For document summarization, hierarchical attention first attends to important sentences, then to important words within them. Coreference resolution relies on attention to link pronouns to their antecedents across paragraphs.
6. Interpretability and Probing
Attention distributions are often treated as explanations for model decisions. Critics argue that high attention weights do not always correlate with feature importance. Nonetheless, probing studies show that specific heads specialize in syntax, rare words, or positional tracking. Understanding these patterns helps diagnose failure modes in large language models.
7. Future Directions
Researchers are exploring recurrent memory mechanisms, state-space models, and linear attention variants that maintain expressive power while scaling linearly with sequence length. Hybrid architectures that combine local and global attention remain promising for long-document understanding and agentic tool use.
Conclusion
Attention mechanisms moved NLP from sequential bottlenecks to global context modeling. The ability to dynamically focus on relevant context remains the defining feature of modern large language models.
"""
print(f"Document loaded: {len(DOCUMENT)} characters")
Step 4: Build the query agent
This function wraps the document and the user question into a single message. Because Oxlo.ai uses per-request pricing, sending the full document every time costs the same regardless of length. No need for chunking logic or token counters.
def ask_document(question: str) -> str:
user_message = f"Document:\n{DOCUMENT}\n\nQuestion: {question}"
response = client.chat.completions.create(
model=MODEL,
messages=[
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": user_message},
],
)
return response.choices[0].message.content
# Quick factual lookup
print(ask_document("Who proposed the first neural attention mechanism?"))
Step 5: Add multi-hop reasoning
A single-hop question only tests retrieval. A multi-hop question forces the model to attend to the introduction, the Transformer section, and the conclusion to synthesize a comparative answer. This is where attention mechanisms prove their value over bag-of-words models.
MULTI_HOP_QUESTION = (
"How did the shift from RNN-based attention to self-attention solve the "
"fixed-length bottleneck mentioned in the introduction, and which section "
"describes the mathematical operation that enables this?"
)
print(ask_document(MULTI_HOP_QUESTION))
Run it
Execute the full pipeline. The single-hop query should return Bahdanau et al. The multi-hop query should connect the bottleneck in the introduction to the scaled dot-product attention formula in Section 3.
if __name__ == "__main__":
print("=== Single-hop ===")
print(ask_document("Who proposed the first neural attention mechanism?"))
print("\n=== Multi-hop ===")
print(ask_document(
"How did the shift from RNN-based attention to self-attention solve the "
"fixed-length bottleneck mentioned in the introduction, and which section "
"describes the mathematical operation that enables this?"
))
Example output:
=== Single-hop === Bahdanau et al. proposed the first neural attention mechanism for sequence modeling (Section 1, Introduction). === Multi-hop === The fixed-length bottleneck in early sequence-to-sequence models limited the decoder to a single compressed representation of the source sentence. Self-attention removed this bottleneck by allowing every token to directly attend to every other token, eliminating the need to fit the entire source into one vector (Section 2). The mathematical operation that enables this parallel global interaction is scaled dot-product attention, defined as softmax(QK^T / sqrt(d_k))V (Section 3, Multi-Head and Scaled Dot-Product Attention).
Next steps
Swap the model ID to kimi-k2.6 or deepseek-v4-flash to test a 131K or 1M context window on an entire book or codebase. Because Oxlo.ai pricing is per request, the cost stays flat even when you feed hundreds of thousands of tokens in one shot. See https://oxlo.ai/pricing for details.
Alternatively, add function calling to the agent so it can pull documents from a URL or query a vector store before falling back to full-context analysis. Oxlo.ai supports tool use on all chat models, so the same client code works with no library changes.
Top comments (0)