DEV Community

Cover image for Codebase Knowledge Base Series (04): Three Chunking Strategies — AST Precision Loses?
WonderLab
WonderLab

Posted on

Codebase Knowledge Base Series (04): Three Chunking Strategies — AST Precision Loses?

#ai #ast #codegraph

The Mistake Almost Everyone Makes

You need to split code into chunks to feed a vector model. What's the most "professional" way to do it?

Most engineers answer instantly: chunk by AST (abstract syntax tree), one function per chunk. The reasoning is airtight — a function is the most natural semantic unit, AST aligns precisely with semantic boundaries, and it never cuts a function in half. By comparison, splitting by fixed line count (say, every 20 lines) looks crude and dumb: it hacks functions into fragments.

Sounds unassailable. But the experimental data slaps that intuition across the face.

On the same 266 lines of Python, I ran three chunking strategies. The result: AST function-level chunking scored the lowest. The "crude and dumb" fixed-line split scored a perfect 1.000.

This isn't random noise. There's a mechanism underneath worth digging into.

The Three Strategies

Dataset: 266 lines of Python covering 5 modules — auth, database, cache, payment, notification — 27 functions total. Embedding model is bge-large-zh-v1.5 throughout; evaluation uses 12 natural-language queries.

Three ways to cut:

Strategy 1: Fixed-line (20 lines each, overlap=3)

Ignore code structure, slice every 20 lines, overlap adjacent chunks by 3 lines to avoid losing boundary info.

Chunk 1: lines 1-20
Chunk 2: lines 18-37   ← overlaps 3 lines with the previous chunk
Chunk 3: lines 35-54
...
Enter fullscreen mode Exit fullscreen mode

Strategy 2: File-level (one chunk for the whole file)

Brute force. The entire file is a single chunk.

Chunk 1: lines 1-266   ← everything crammed in
Enter fullscreen mode Exit fullscreen mode

Strategy 3: AST function-level (one chunk per function)

Parse the syntax tree with Python's ast module, split precisely on function definitions.

Chunk 1: def validate_jwt_token(...)    lines 9-18
Chunk 2: def hash_password(...)         lines 20-28
Chunk 3: def create_payment_intent(...) ...
...
Enter fullscreen mode Exit fullscreen mode

First, let's see exactly how "crude" fixed-line chunking is. Take validate_jwt_token (lines 9-18, 10 lines total):

Function 'validate_jwt_token' (lines 9-18, 10 lines)
Splits across 2 chunks:
  Chunk lines 1-20:  contains lines 9-18 (10/10 lines of function)
  Chunk lines 18-37: contains lines 18-18 (1/10 lines of function)

Problem: Neither chunk contains the complete function.
A query about 'validate_jwt_token' will retrieve a partial implementation.
Enter fullscreen mode Exit fullscreen mode

See it? The function's last line (line 18) got split into the next chunk. This is exactly what fixed-line chunking is criticized for — it doesn't know where function boundaries are, and it cuts wherever it wants.

By all rights, that fragmentation should hurt retrieval. AST keeps every function whole. It should win, no contest.

The Results 

Strategy                      Chunks   AvgLen      R@3      R@5  vs AST R@5
──────────────────────────── ───────  ───────  ───────  ───────  ──────────
1_fixed_lines_20                  16      755    0.917    1.000      +0.042
2_file_level                       1    10270    1.000    1.000      +0.042
3_ast_function                    27      356    0.889    0.958        base
Enter fullscreen mode Exit fullscreen mode

The reversal happened:

  • Fixed-line: Recall@5 = 1.000 (perfect)
  • File-level: Recall@5 = 1.000 (perfect)
  • AST function-level: Recall@5 = 0.958 (lowest)

The "crude method" that chops functions apart won. The "professional method" that aligns precisely with semantic boundaries lost.

Of the 12 queries, 11 scored a perfect 1.000 across all three strategies. The only divergence was query 8:

Query                                               Fixed    File     AST
────────────────────────────────────────────────── ──────  ──────  ──────
process payment and create Stripe charge             1.00    1.00    0.50 ←
(the other 11 queries all score 1.00 across strategies)
Enter fullscreen mode Exit fullscreen mode

One query decided it. On this one, AST got 0.50, while fixed-line and file-level both got 1.00.

To understand the reversal, we have to understand exactly what happened on this single query.

The Free-Rider Effect

The query is process payment and create Stripe charge.

Its ground truth has two relevant functions: create_payment_intent and calculate_order_total.

  • create_payment_intent contains tokens like stripe.PaymentIntent and create() — tightly aligned with "Stripe charge." All three strategies hit it easily.
  • calculate_order_total is the problem. Its job is "sum item prices, apply discount, compute tax" — its signature and body are full of sum, discount, tax, with no trace of "payment" or "Stripe." In vector space, it sits far away from the query "create Stripe charge."

Here's the crucial difference:

AST chunking: calculate_order_total is cut into its own isolated chunk, standing alone in vector space. It's far from the query, so it isn't retrieved — AST scores 0.50 (1 of 2 hit).

Fixed-line / file-level chunking: calculate_order_total and create_payment_intent happen to land in the same chunk (they're adjacent in the source). When the query hits create_payment_intent, the whole chunk gets retrieved, and calculate_order_total comes along for the ride — the free-rider effect.

[Image: Vector space diagram. Left, AST chunking: calculate_order_total is an isolated point far from the query, not retrieved. Right, file/fixed chunking: calculate_order_total and create_payment_intent are boxed together; when the query hits the box, both functions are pulled out.]

An analogy:

AST chunking is like a supermarket where every item sits on its own shelf with its own barcode. Search "beer," you only find beer.
A large chunk is like bundling beer and diapers into one promo pack. Search "beer," hit the pack, and diapers land in your cart too — even though you never searched for diapers.

On this tiny 266-line dataset, the "bundles" happened to pair up semantically adjacent functions correctly, so they piggybacked in the otherwise-unfindable calculate_order_total. AST, by cutting so cleanly, lost that "accidental win."

That's the whole truth of the reversal: AST chunking isn't worse — the large chunks got lucky on one query via the free-rider effect.

Don't Hand Fixed-Line the Trophy Yet

It's tempting to conclude: "Just use fixed-line/file-level from now on, done." Hold on — that conclusion is dead wrong. The reason hides in a column the table conveniently omits.

File-level chunking has a perfect Recall@5, but its precision is zero.

Think about how file-level works: the entire codebase is 1 chunk. Any query retrieves the same chunk — all 266 lines. Recall@5 measures "is the right answer in the top 5," and since there's only 1 chunk containing every function, of course it always hits.

But what good is that? Ask "where's JWT validation," and it hands you the entire file. You still have to hunt through 266 lines yourself. The point of a retrieval system is localization, and file-level chunking localizes nothing. Its perfect score is cheating.

Fixed-line isn't so sweet either. This dataset's functions are short and dense, with a small average length. A 20-line chunk plus 3-line overlap happens to fit many functions whole into a single chunk — even letting adjacent functions free-ride. That's a coincidence of dataset characteristics meeting chunk size, not a virtue of fixed-line chunking.

Try it on a real large project: functions routinely run over a hundred lines, and a 20-line chunk can't hold even half a function. The validate_jwt_token fragmentation from the opening demo becomes the norm — you always retrieve function fragments, with the signature in one chunk, the core logic in another, exception handling in a third. At that point, fixed-line Recall collapses.

This experiment's scale (266 lines) is too small — it happens to mask the precision differences, making the two "cheaters" look like winners.

When Each Strategy Actually Fits

Factor in scale, and the picture clears up:

Small codebase (< 1000 lines)
File-level is passable — cramming it all into an LLM's context window is cheap anyway. But be clear that its precision is zero; it's essentially "punting retrieval to the LLM." Good for the "code small enough to hand the whole thing to the model" scenario.

Medium codebase (1K - 10K lines)
Fixed-line + reasonable overlap usually suffices, and it's the simplest to implement (no AST parsing). The key is matching chunk size to your average function length — ideally letting most functions land whole in one chunk, with overlap as a boundary safety net.

Large codebase (> 10K lines)
This is where AST function-level's precision advantage finally shows. When the code is too large to fit in context, you need to pinpoint "this exact function," and every AST chunk is a complete, independent, jump-to-able semantic unit. This experiment's dataset is too small — it completely masks AST's core strength.

In one line: Recall looks flashy, but in production, precision is the metric that decides the experience — and precision is exactly what this small experiment can't measure.

Best Practice: AST + Metadata + Call Graph

So what's the right answer for large projects? Not picking one of the three strategies — it's using AST chunking for precision, then actively restoring the free-rider effect via metadata and the call graph.

Recall where AST lost: it cut calculate_order_total into an island, discarding the context that "it's in the same payment flow as create_payment_intent." Large chunks preserved that context by physical adjacency, by luck. We can add it back explicitly:

{
    "content": func_body,                    # complete function body from AST, fed to embedding
    "metadata": {
        "name": "calculate_order_total",
        "module": "payment",                 # module name: enables per-module filtering
        "file": "services/payment.py",
        "line_start": 42,
        "line_end": 55,
        "called_by": ["create_payment_intent"],  # call graph: who calls me
        "calls": ["get_tax_rate", "apply_discount"],
    }
}
Enter fullscreen mode Exit fullscreen mode

With the called_by edge, the retrieval flow can recover like this:

  1. Query process payment and create Stripe charge; vector search hits create_payment_intent.
  2. Follow the call graph, discover create_payment_intent calls calculate_order_total.
  3. Recall calculate_order_total along with it.

This uses the deterministic structural edge of the call graph to actively reproduce the free-rider effect that large chunks only got by luck — without sacrificing AST's precision. Functions that are semantically related but distant in vector space get pulled back by their call relationships.

This is exactly where vector retrieval hits its ceiling: the semantic gap between calculate_order_total and "Stripe charge" cannot be closed by any embedding strategy (a conclusion Article 03 already validated). To cross that gap, you need code's own structural information — the call graph, imports, module membership.

And that's precisely the topic of the next article in this series: Vector Retrieval vs. Knowledge Graph. When semantic similarity fails, a code's structural relationships are the lifeline.

Summary

  1. Counterintuitive result: AST function-level chunking has the lowest Recall@5 (0.958); fixed-line and file-level both score a perfect 1.000. But this "loss" is an illusion.
  2. The decider is the free-rider effect. On the only divergent query (Q8), calculate_order_total is semantically far from the query; AST cuts it into an island and misses it, while large chunks bundle it with create_payment_intent and free-ride it back.
  3. File-level's perfect score is cheating — its precision is zero. With just 1 chunk, it can't pinpoint a specific function; it's essentially punting retrieval to the LLM.
  4. Fixed-line's perfect score is coincidence. The dataset's functions are short and dense, happening to match the chunk size. On a large project with hundred-line functions, fixed-line shreds functions into fragments.
  5. Scale decides the choice: file-level for small, fixed-line + overlap for medium, AST for large. This 266-line experiment is too small and masks the real precision differences.
  6. Best practice: AST function-level + metadata + call graph. Use the call graph's deterministic structural edge to actively restore the free-rider effect that large chunks only got by luck, while keeping AST's precision.

References

Check out PrimeSkills — a curated marketplace of AI agents and skills that have been validated in real-world, enterprise-grade workflows. No fluff, just what actually works.

Find more useful knowledge and interesting products on my Homepage

Top comments (0)