Reviewing Entire Repositories with 32k-Token LLM Context

The Repository Context Problem

When you try to get an LLM to review a whole repository, the token limit feels like a wall. If you only feed the model one file at a time, it misses the architectural patterns and dependencies that exist across your entire codebase.

In this article, you'll learn:

• How to split code into manageable chunks while keeping context.
• Which prompt strategies keep the model focused on the right parts.
• The trade-offs between speed, cost, and accuracy.
• Common failure modes and how to detect them.

Why Long Context Matters

Large codebases contain patterns that only appear across many files. A model that sees only a single snippet may miss a critical dependency or a specific naming convention used in a different directory.

Extending the context window—the amount of text a model can process at once—lets the model see the whole picture. This is vital for tasks like finding security vulnerabilities or refactoring code for better consistency across a project.

Choosing a Strategy

You can handle long text in three main ways depending on your goals. Each approach has a different impact on how much the model "understands" your project.

Approach	Tradeoff	When to use
Sliding Window	Fast and simple, but lacks global view	Finding local bugs or syntax errors
Hierarchical Summaries	Slower and more complex, but captures architecture	High-level design reviews and refactoring
Retrieval-Augmented Generation (RAG)	Highly targeted, but requires an index	Answering specific questions about a large codebase

Chunking and Sliding Windows

A simple function can split text into chunks that fit the model’s token limit. I recommend using an overlap between chunks. This overlap ensures that if a function definition is cut in half, the model sees enough of the context in both chunks to understand what happened.

This function splits text into chunks based on a word count to approximate tokens.

def chunk_text(text, max_tokens, overlap=200):
    # We split by whitespace to approximate token counts
    tokens = text.split()
    chunks = []
    start = 0

    while start < len(tokens):
        # Calculate the end of the current chunk
        end = min(start + max_tokens, len(tokens))
        chunk = " ".join(tokens[start:end])
        chunks.append(chunk)

        # Move the start pointer forward, subtracting overlap
        start += max_tokens - overlap

        # Break if we've reached the end of the text
        if end == len(tokens):
            break

    return chunks

I use a simple whitespace split here because it's easy to reason about. In a production environment, you'll want to use a library like tiktoken to count actual tokens, as one word doesn't always equal one token.

Implementing a 32k Token Pipeline

Below is a minimal example that reads a repository, chunks the code, and sends each chunk to the model. This approach is useful for a first pass of a codebase.

import os
import openai

def load_repo(repo_path):
    code = ""
    # We walk the directory tree to find relevant files
    for root, _, files in os.walk(repo_path):
        for f in files:
            if f.endswith((".py", ".js", ".ts")):
                with open(os.path.join(root, f), "r", encoding="utf-8") as fp:
                    code += fp.read() + "\n"
    return code

def review_code(repo_path, model="gpt-4o-mini"):
    code = load_repo(repo_path)
    # We use a chunk size slightly smaller than the limit to be safe
    chunks = chunk_text(code, max_tokens=30000, overlap=500)

    for i, chunk in enumerate(chunks):
        prompt = f"Review the following code for logic errors:\n{chunk}"
        response = openai.ChatCompletion.create(
            model=model,
            messages=[{"role": "user", "content": prompt}],
            temperature=0.2,
        )
        print(f"Chunk {i+1} review:\n", response["choices"][0]["message"]["content"])

This script is a starting point. In a real-world tool, you'd need to handle API rate limits and add error handling for files that can't be read.

Common Failure Modes

Even with a large context window, things can go wrong. You should watch out for these common issues:

• Context loss: If your overlap is too small, the model might lose the connection between a variable declaration and its usage.
• Token budget overflow: If your chunks are too large, the API will return an error. Always leave a buffer.
• Model hallucination: If the prompt is too vague, the model might invent bugs that don't exist. Use specific instructions.
• Cost spikes: Sending many large chunks can get expensive quickly. Monitor your usage.

Key Takeaways

• A 32k token model lets you review significant portions of a repository without manual splitting.
• Chunking with overlap preserves context across file boundaries.
• Prompt consistency reduces hallucinations and makes the output easier to parse.
• Always use a token-counting library rather than counting characters or words for precision.

Source

AI has access to a vastly larger working memory than the human brain — I added working code, a comparison table, and failure-mode analysis.