Stop Losing Chunk Context: Anthropic's Contextual Retrieval with Spring AI and Virtual Threads

Stop Losing Chunk Context: Anthropic's Contextual Retrieval with Spring AI and Virtual Threads

Naive chunking destroys document semantics, causing over half of production RAG retrieval queries to miss critical context. Prepending dynamic contextual summaries via Anthropic’s prompt caching—executed concurrently across chunks using Java Virtual Threads—fixes this without blowing your latency or LLM budget.

Why Most Developers Get This Wrong

• Relying on naive splitters: Splitting strictly by token limits or arbitrary overlaps yields orphan chunks like "EBITDA dropped 4%," with zero metadata indicating the company or fiscal quarter.
• Sequential LLM enrichment: Processing chunks sequentially against an LLM turns document ingestion pipelines into a multi-hour operational bottleneck.
• Ignoring prompt caching: Re-sending the entire parent document for every single chunk annotation burns thousands of input tokens and bankrupts your RAG budget.

The Right Way

Inject LLM-generated chunk context using Claude 3.5 Sonnet and prompt caching, fanned out concurrently via Java Virtual Threads into a hybrid BM25 and vector index.

• Enable prompt caching on the parent document so all chunk-enrichment calls read the large parent context from cache at an 80-90% discount.
• Fan out chunk-enrichment calls concurrently inside a Spring AI DocumentTransformer using an unbounded Virtual Thread executor.
• Format chunks as [Context: {summary}]\n\n{original_text} before generating embeddings and BM25 lexical tokens.
• Query your vector store (e.g., PgVector) using Spring AI's hybrid search API to capture both semantic similarity and exact keyword anchors.

Show Me The Code

public class ContextualTransformer implements DocumentTransformer {
    private final ChatModel chatModel; // Configured for Claude 3.5 Sonnet

    public List<Document> apply(List<Document> chunks, Document parentDoc) {
        try (var executor = Executors.newVirtualThreadPerTaskExecutor()) {
            var futures = chunks.stream().map(chunk -> executor.submit(() -> {
                String prompt = """
                    <document>%s</document>
                    Situate this chunk within the parent document in 2-3 concise sentences:
                    <chunk>%s</chunk>""".formatted(parentDoc.getText(), chunk.getText());
                String context = chatModel.call(prompt);
                return chunk.mutate().text(context + "\n\n" + chunk.getText()).build();
            })).toList();
            return futures.stream().map(Future::join).toList();
        }
    }
}

Key Takeaways

• Naive chunking is obsolete; Contextual Retrieval eliminates chunk ambiguity by embedding parent-document context directly into the text.
• Virtual Threads give you non-blocking, massive LLM concurrency in Java without the mental overhead of Reactive Streams.
• Prompt caching makes parallel contextual ingestion economically viable by slashing input token costs on parent documents.

Shameless plug: javalld.com has full LLD implementations with step-by-step execution traces — free to use while prepping.