Maya Richardson

Posted on Sep 26

Context Engineering: How to Stop Wasting Tokens on Long-Context LLMs

#llm #ai #context #developers

A million tokens is a bucket, not intelligence. The hard part isn't having room for data—it's deciding what goes in.

The mistake most engineers make first is treating long-context as a storage solution. You get excited about a 1-million-token window, paste in every document, every log, every config file, then wonder why the model hallucinates or misses obvious answers. On the OpenAI Multi-Needle Context Retrieval benchmark, accuracy dropped 10.9 percentage points between 128,000 and 1 million tokens. The problem isn't the model; it's that you're asking it to do curation work before it ever sees your actual question.

Prerequisites

Familiarity with API calls to an LLM (I'm using Python and curl examples, but adapt to your stack).
Access to an LLM with a long context window (GPT-4.1, Gemini 2.5 Pro, or Claude Sonnet 4 on Bedrock all support 1 million tokens).
A real use case with multiple documents, logs, or source files—not a toy example.
Basic understanding of what your model needs to answer your question (this is the critical part).

1. Audit what's actually relevant

Before you touch an API, sit with your use case and ask: what information does the model need to answer correctly?

If you're asking a question about a specific customer's issue, you don't need the entire customer database. If you're debugging a production error, you don't need every log from the past six months. You need the signal.

Take 15 minutes and list the documents, sections, or data types that directly answer your question. Order them by importance. Be ruthless about cutting anything that's nice-to-have but not necessary. This is the foundation.

2. Chunk and tag documents by domain

Once you know what's relevant, break it into small, labeled pieces. A document with a title, a summary, and metadata (like creation date or category) is much easier for the model to navigate than a massive undifferentiated dump.

The why: models process text sequentially. Chunking and tagging let you hint at structure. One study found that dynamically generated distractions caused an average performance drop of more than 45% across mainstream models. Irrelevant context actively works against you.

The trade-off: you're doing extra preprocessing work upfront. But it's cheaper (in tokens and latency) than letting the model struggle.

Here's how I'd structure it:

import json
from datetime import datetime

# Example: chunking customer support tickets
tickets = [
    {
        "id": "TKT-2025-001",
        "date": "2025-01-15",
        "domain": "billing",
        "summary": "Customer reports duplicate charge on recurring subscription",
        "body": "Customer was charged twice for monthly plan on Jan 15. Refund issued. Root cause: payment processor retry logic fired twice due to network timeout."
    },
    {
        "id": "TKT-2025-002",
        "date": "2025-01-16",
        "domain": "api_access",
        "summary": "Rate limiting causing 429 errors in batch processing",
        "body": "Client hitting rate limits when submitting 500 requests in parallel. Solution: implement exponential backoff and request queuing."
    }
]

# Tag and serialize for the prompt
context_items = [
    f"[{t['domain'].upper()}] {t['summary']}\nID: {t['id']} | Date: {t['date']}\n{t['body']}"
    for t in tickets
]

print("\n---\n".join(context_items))

The figures this piece relies on — sources: openai.com.

3. Use retrieval or filtering before the prompt

Now that you have tagged, chunked context, don't pass all of it to the model. Filter or retrieve only the pieces that match your question.

The why: even with 1 million tokens, you're paying for every token you send (and models get worse at finding needles as context grows). If your question is about a billing issue, retrieve only the billing-tagged chunks. This is called context engineering, and it's the difference between a working system and an expensive one.

The trade-off: you need a retrieval mechanism—semantic search, keyword matching, or a lightweight embedding model. It adds one step. But it cuts token spend and latency, and improves accuracy.

Here's a simple keyword-based filter:

def filter_context(items, query_keywords, domain=None):
    """Filter context items by keyword and optionally by domain."""
    filtered = []
    query_lower = query_keywords.lower()

    for item in items:
        # Match domain if specified
        if domain and f"[{domain.upper()}]" not in item:
            continue
        # Match keywords in summary or body
        if any(kw in item.lower() for kw in query_lower.split()):
            filtered.append(item)

    return filtered

# Example usage
query = "duplicate charge billing"
relevant = filter_context(context_items, query, domain="billing")
print(f"Filtered to {len(relevant)} items")

For production, use embeddings or a vector database. The principle is the same: retrieve, don't broadcast.

4. Construct a context hierarchy: tier by importance

Some context is critical; some is supporting. Arrange it so the model sees the critical pieces first and can skip the rest if needed.

The why: even with filtering, models often attend to the early tokens more strongly than late ones. If your most important context arrives first, it has more influence on the answer. Ryan Peters, Director of AI Research at Heisenberg Research Labs, puts it this way: "A million-token window is a bucket, not a brain—if you pour in everything hoping the model will sort it out, you're asking it to do the curation work you should have done before the prompt ever left your keyboard."

The trade-off: none, really. This is pure structure, and it costs nothing.

Frame it like this:

critical = relevant[0] if relevant else "No critical context found"
supporting = "\n---\n".join(relevant[1:]) if len(relevant) > 1 else "No supporting context"

context_prompt = f"""You are answering a question about customer support tickets.

## CRITICAL (use this first):
{critical}

## SUPPORTING (reference if needed):
{supporting}

## QUESTION:
{query}

Answer based on the critical context first. Use supporting context only if it adds clarity.
"""

5. Watch for irrelevant distractions; prune aggressively

The moment you start seeing the model second-guess itself or contradict your known facts, something in the context is working against you—a conflicting log entry, an old policy, outdated docs.

The why: models struggle with contradictions. Irrelevant context actively hurts accuracy. On a reasoning task with five reasoning steps, one study found that GPT-4.1's accuracy fell from 26% with one irrelevant context to 2% with 15 irrelevant contexts.

The trade-off: you have to maintain discipline. It's tempting to throw more in; it's hard to leave things out. But leaving it out is the right move.

Test: if removing a document doesn't change or worsen your answer, it stays out.

6. Cache repeated context to cut latency and cost

If you're using the same context across multiple questions (like a codebase, a knowledge base, or a client profile), cache it. OpenAI said GPT-5.1 extended prompt caching to 24 hours, with cached input tokens 90% cheaper than uncached tokens; other providers offer similar discounts on repeated prefixes.

The why: the provider reuses the work it already did on an identical prompt prefix instead of processing it again. For long context, that saves both money and time to first token.

The trade-off: caches expire (24 hours on GPT-5.1; on Anthropic's API, used in the example below, five minutes by default with an optional one-hour lifetime), and only an identical prefix hits the cache. Put the static material first and the changing question last, and decide how you refresh the static part when it changes.

import anthropic

client = anthropic.Anthropic()

# Static context (e.g., codebase, docs) that repeats across requests
static_context = """
# Our API v2 Reference
...[large static doc]...
"""

response = client.messages.create(
    model="claude-sonnet-5",
    max_tokens=1000,
    system=[
        {
            "type": "text",
            "text": "You are a helpful API documentation assistant."
        },
        {
            "type": "text",
            "text": static_context,
            "cache_control": {"type": "ephemeral"}
        }
    ],
    messages=[
        {"role": "user", "content": "How do I authenticate?"}
    ]
)

print(response.content[0].text)

Common mistakes

Dumping everything and expecting the model to filter. You're asking the model to do your job. The model is for reasoning over context you've already curated.

Ignoring latency. GPT-4.1 took approximately 1 minute to return a result at 1 million tokens versus approximately 15 seconds at 128,000 tokens. That's a hard wall. If your use case needs speed, don't pretend a million-token window is free.

Mixing old and new data. If your context includes both today's logs and logs from six months ago, the model can get confused about what's current. Use dates, version tags, or separate old context into an explicit "reference" tier.

Not measuring. Send the same question with different amounts of context and log the accuracy. If adding more doesn't improve it, you've found your signal-to-noise threshold. Stop there.

FAQ

Q: When should I actually use the full long-context window?

A: When the answer genuinely depends on synthesizing information from many independent sources and you've already filtered ruthlessly. Example: summarizing a long legal document or correlating events across multiple logs. If your use case doesn't involve that kind of synthesis, you're probably over-using it.

Q: How do I know if my context is too much?

A: Test it. Remove a chunk. Re-run the query. If the answer is the same, the chunk wasn't needed. If accuracy drops when you remove something, keep it. That's the only honest test.

Q: Should I always use semantic search / embeddings for context retrieval?

A: Start with keyword filtering and domain tagging. If that works, stick with it—it's simpler and cheaper. Move to embeddings only if keyword matching leaves you with too many false positives or false negatives. Most use cases don't need embeddings until they hit scale.

DEV Community