Lê Tú Hào

Posted on Aug 9

AI Engineering #02 — Chunk-First, Chunk-Native: The Unit of Work Is a Data-Model Decision, Not a Prompt-Time Patch

#architecture #software #ai #learning

Previous: #01 — When an AI Discards Its Own Search Results

This is a post-mortem on my own system, not someone else's. And here's the uncomfortable part: I knew the books could be enormous — it was written into the project's own goals, in black and white, before a line of code. I just didn't treat that knowledge as a design constraint. I filed it under "handle it later," built for the common case, and let every layer quietly hardcode "the whole thing fits." Then I pointed it at a real 4,000+ chapter web-novel and every feature broke at once. The lesson — that knowing isn't designing — cost me over a month of refactor work, and I'm still paying it down today. I'm writing it down so you can buy it for the price of reading instead.

The short version

I treated chunking — splitting a large input into smaller units — as something you do at the LLM call, right before you hit the context window: get the document, realize it won't fit, slice it into pieces, loop. Problem solved. I doubt I'm the only one who started there — but I'll argue from my own system, not from a statistic I don't have.

It isn't solved. It's deferred, and the interest compounds.

By the time you're slicing text at the prompt boundary, the rest of your system has already committed to the opposite assumption. The document is one row in your database, one job in your queue, one request to your API, one entry in your cache, one item in your UI list. Chunking at the LLM call fixes the prompt and leaves every other layer believing the whole thing still fits. So the whole thing breaks — not at the model, but at the database, the job runner, the API, the frontend — the first time an input is genuinely large.

The fix is to treat what your unit of work is as a foundational decision — and to keep that decision reversible, so no layer hardcodes "the whole document" in a way you'd have to migrate out of later. I'll argue for two ideas, deliberately a pair. The two labels are mine — not established industry terms — and they're names for older engineering ideas I'm connecting, not new primitives:

Chunk-first — the timing lesson. Decide the grain of your unit of work early, and give it an identity in your schema, before the rows, clients, and caches accumulate that make it expensive to change.
Chunk-native — the structural lesson. As scale arrives, every layer that does work — storage, jobs, caching, APIs — operates on the chunk, not the document. Like cloud-native: the property pervades the stack instead of living in one function.

   DOCUMENT-NATIVE   (the default — and what breaks at scale)
   ┌───────────────────────────────────┐
   │           WHOLE  DOCUMENT         │ ─── one row · one job · one request ·
   └───────────────────────────────────┘       one cache key · one prompt
                                          only the prompt ever gets split — the trap

   CHUNK-NATIVE   (the chunk carries the work; the document organizes it)
   document = container   (an ordering + a manifest + the ownership boundary)
   ┌──────┬──────┬──────┬──────┬──────┐
   │ chunk│ chunk│ chunk│ chunk│  ... │ ─── the CHUNK is the unit of work:
   └──────┴──────┴──────┴──────┴──────┘      one row · one job · one request ·
      each chunk has: id · hash · version      one cache key · one prompt (per chunk)

This is the whole essay in one line. Chunk-first does not mean "build the chunk-native stack up front," and it does not mean "pick the perfect grain on day one." It means make the one cheap decision that keeps the seam open: give your unit of work its own identity in the schema. Get that right, and going chunk-native later is a migration whose cost you control. Skip it, and it's a migration you pay for all at once, on the schedule your largest input picks for you.

One disambiguation, because the word is overloaded. If you work in RAG, "chunking" almost certainly means retrieval chunking — splitting text so embeddings retrieve well (fixed-size, recursive, semantic, late chunking). That is not what this post is about. Retrieval chunking is one chunk boundary, at one layer. I mean chunking in the older, data-engineering sense: the grain of the unit your whole system operates on — the database row, the queue job, the API item, the cache key. Retrieval chunking is a special case. When I say chunk-first, I mean the unit-of-work decision, not the embedding-window decision.

One document can carry several grains at once, and they do not have to agree:

DOCUMENT (a book)
   ├── processing grain   → a scene      (one extraction job, one cache key)
   ├── persistence grain  → a scene      (one row, one id)
   ├── job grain          → a scene      (one checkpoint)
   ├── meaning grain      → a chapter    (what the user names and owns)
   └── retrieval grain    → ~400 tokens  (what embeds well)

The RAG conversation only ever discusses the last line. This post is about the other four.

One honesty note before you start: this is one system in one domain. Treat the claims as a strong hypothesis shaped by long, mutable, user-supplied text — not a proven law.

Contents — and where to start, depending on who you are

Where to start. If you already think in declare-the-grain, durable-execution, and semantic-operator terms, you know the mechanics — skim to None of this is new, the one section written for you, and grab the retrofit playbook. If you're a RAG engineer who's only ever chunked for retrieval, read it all: the load-bearing claim is that "chunking" names two different decisions and you've made only one. Early-career? Jump to How to tell you built document-native and use it as a checklist.

The four failure modes (and the one decision)
The principle: chunk-first & chunk-native
When it applies, and how to retrofit
None of this is new
Closing
Appendix — the lore-weave case study, in depth
Prior art and further reading

The four failure modes — and the one decision underneath

Start with the problem, in general terms. Every system that processes large inputs begins with one reasonable-looking assumption: the document is self-sufficient — it fits in one unit of work (one row, one job, one request, one prompt), and it carries its own organization, so nothing else needs storing. That assumption is invisible on small data and load-bearing on large data, and it fails in four recognizable ways as inputs grow. Here they are as general patterns; if you've built anything that ingests user-supplied documents, you've probably met at least one.

The failure mode — does your system do this?	The fix
The unbounded unit. An input that looks bounded — "one document" — but whose real payload is the document plus the context the operation needs, and that context grows with your data.	Process the sub-unit, not the document; feed each call only the slice of context it needs.
The whole-dataset operation in one request. A step that scans or rebuilds your entire dataset synchronously inside a single request — invisible on demo data, fatal at scale.	Bound it by construction: range-scope it, cap it, make it async.
The all-or-nothing job. One document = one long-running job with no checkpoint, so a crash near the end discards all the work before it.	Make the sub-unit the unit of work; checkpoint + resume; key each by a content hash so re-runs are free.
The discarded structure (the container, not the atom). The durable structure that organizes your data is thrown away once the derived output exists, so you can't rebuild, diff, or reconcile it.	Keep the structure as first-class, diff-able data you can rebuild from.

Four separately-named bugs — a missing bound, a missing index/split, a missing checkpoint, a discarded source of truth — and each is caught, in isolation, by ordinary competent engineering. The reason to name them together is that in my system they shared one upstream assumption: that a document is self-sufficient. That assumption has two halves, and they fail differently:

"It fits." One document is one unit of work — one row, one job, one request, one prompt. Failures 1–3 are this half breaking. They're about the atom: the unit was too big, unbounded, or un-checkpointed.
"It describes itself." The document implicitly carries everything needed to reconstruct its own organization, so the structure that produced it doesn't need storing. Failure 4 is this half breaking. It's about the container.

Each of these bugs has other causes too — you can write an unbounded backfill in a perfectly chunk-native system, a missing index is a missing index, and you can discard your outline at any size. The claim isn't that this assumption is the only source of these four. It's that when you find one of them, the assumption is worth checking, because it tends to have produced the others as well. Fix them one at a time and you treat symptoms; name the shared assumption and you find the rest before they fire.

So one decision sits underneath all four: what is your unit of work — and does it have its own identity in your schema? The rest of this post is that decision — how to make it and how to retrofit if you're already stuck. A detailed case study — a real 4,000-chapter book that sprang every trap above, with the quotes and numbers — is in the appendix for those who want the receipts.

If that table is all you read, you have the point. The rest is proof and procedure.

The principle: chunk-first (when) and chunk-native (how)

Count the layers that assumed one document = one unit

Walk a request from the UI down to the model, and notice how the "document = one unit" assumption is quietly baked into every layer — so a failure can surface at any of them:

      a request flows this way  ─────────────────────────────────────────>
   ┌────────┐ ┌────────┐ ┌────────┐ ┌────────┐ ┌────────┐ ┌────────┐
   │  UI    │→│  API   │→│  JOB   │→│  DB    │→│ CACHE  │→│  LLM   │
   │  list  │ │ request│ │ queue  │ │  row   │ │  key   │ │  call  │
   └────────┘ └────────┘ └────────┘ └────────┘ └────────┘ └────────┘
   "one book" "one call" "one task" "one row"  "one key"  "fit it all"
       ^                     ^          ^                     ^
       │                     │          │                     │
    list returns          job = 1     whole-dataset       oversized
    everything            doc, no     scan, in one        prompt
    (no paging)           resume      request             (first symptom)

You "add chunking" at the far right (the prompt) and fix the rightmost box. Every box to its left still hands you the document whole. So you retrofit: the DB row becomes a row per chunk (schema migration + backfill); the job becomes N jobs (idempotency, checkpointing, partial-failure handling); the API grows pagination (a contract change every client must follow); the cache re-keys per chunk; the UI grows a pager. Each of those is the tax I paid — and I paid it after the schema, the stored data, and the callers already existed, which is the expensive time to change any of them.

Concretely, the bill: over a month of refactor work, and I'm still paying it — the finest split stayed on my backlog long after the rest had shipped. It wasn't a rewrite; I did it incrementally, in place, without stopping feature work. Retrofitting is survivable. But none of that month bought a single new feature. Every hour went to undoing an assumption I could have declined to make in an afternoon, and the price compounds with every row you migrate and every caller you have to update. That's the real shape of the cost: not a wall you hit once, but a tax you keep paying because the design was made without awareness.

The unit of work is a first-class design object

Here's the reframe that would have saved me. Before you model the domain, answer one question:

What is the smallest piece of this input that a single operation must act on — and does that piece deserve an identity?

If the answer is "a scene," or "a passage," or "a 500-token span," then that is your row, your job, your cache key — not the document that contains it. The document becomes a container (an ordering, a parent id, a manifest, the ownership boundary), and the chunk becomes the atom of work.

Read "atom" narrowly: the atom of expensive operations, not of everything. One document legitimately carries several grains at once — a scene for extraction, a 400-token span for retrieval, a chapter for the user's mental model — and they don't have to agree. The rule I land on later is chunk-granularity where the work happens; document structure where the meaning lives. This post is not an argument that everything becomes a chunk. It's an argument that every expensive operation should have an explicit, bounded processing grain, and that grain should be representable in the data model.

This isn't new, and the oldest name for it is the best: in dimensional modeling it's "declare the grain" — Kimball's rule that you fix the grain of a fact table before you choose its dimensions or measures. That is, almost word for word, "decide the unit of work before you design the schema" — from the 1990s. (Two terminology caveats. I'm borrowing Kimball's grain by analogy: his is the meaning of one fact-table row in a dimensional model; mine is the smallest independently processable unit of a pipeline. The bridge is real — declare it early, declare it precisely — but the extension to processing grain is mine, not his. And I use "unit of work" loosely; in Fowler's Patterns of Enterprise Application Architecture "Unit of Work" is a specific pattern about transaction bookkeeping, not record grain. Read my "unit of work" as "the grain.")

A first-class chunk needs three properties the document-native version never gives it:

Stable-enough identity, under a stated reconciliation policy. A chunk needs an id that downstream references — a graph fact, a translation, a citation — can point at without dangling when the author fixes a typo three chapters over. My approach: on re-processing, match new pieces to old ones by (parent, position) and reuse the id when the text is unchanged. This is surrogate-key and 1NF thinking, applied to text.

Be precise about what that buys you, because I wasn't. Position-matching is stable under in-place edits and cascades under insertion and deletion. Insert a scene at position 2 and every later position shifts: old scene B now sits at 3, so the text at each position no longer matches the id recorded there, and ids churn down the rest of the chapter — the exact failure the scheme was meant to prevent. If your sources get edited structurally rather than just corrected in place, position-matching is not enough, and you want content-based alignment (match on hash first, fall back to position) or author-assigned anchors. "Stable identity" is a policy you choose and state, not a property you get for free.

It also helps to notice that "identity" is three different things that get conflated:

Kind	Key	Answers
Logical / structural	surrogate id, reconciled on re-parse	"is this the same scene as before?"
Content	`hash(text)`	"did the text change?"
Processing	`hash(text, op, model, prompt_version, params)`	"must I recompute this?"

They change at different rates, and conflating them is why my cache and my references disagreed with each other more than once. A scene keeps its logical id through a typo fix; its content and processing identities both change.

Provenance & freshness. Each chunk should know what version of the source it came from and whether it's now stale. This is where I did the most work and, it turns out, reinvented the most: a mutable source that must re-derive only what changed is Incremental View Maintenance / Change Data Capture — materialized views, differential dataflow, a deep and decades-old field. I rebuilt a crude materialized-view refresh with content hashes because the RAG-chunking writing never points you there — not because the problem is unsolved. If your corpus is mutable, go read that literature before you hand-roll it like I did.
Idempotent (really: memoized) processing. Processing a chunk should be a content-addressed, replayable function, so you can checkpoint, resume, parallelize, and cache. Two honest caveats the durable-execution world will insist on: this is memoization, not idempotency in the "apply-twice-equals-once" sense; and an LLM above temperature 0 isn't a pure function, so the cache gives a stable answer, not a faithful replay — and the key must include the prompt template and sampling params, not just the model version.

The chunk-native checklist: where chunking has to live

Chunk-native means every layer that does work operates on the chunk. But not every layer should: ownership, ordering, billing, and the user's mental model stay document-scoped — a reader owns "a book," not "scene #4,812." The rule is: chunk-granularity where the work happens; document structure where the meaning lives. Force the chunk into the meaning layers and you get reassembly chattiness — the N+1 query again, this time of your own making.

Layer	Document-native (breaks)	Chunk-native — do this
Data model	one row per document	one row per chunk; document is a parent / manifest (at document scale a row is the right form — at event scale the representation differs, the identity requirement doesn't)
Identity	document id	chunk id reconciled on re-parse by a stated policy (hash-first, position as fallback)
Ingestion	parse the whole doc in one pass	per-chunk, resumable, hash-keyed upsert
Jobs	one job per document	one job per chunk (or bounded batch); checkpoint + resume; capped in-flight concurrency + backpressure
Caching	`key = doc_id`	`key = hash(chunk, op, model, prompt_version, params)`
LLM calls	fit the doc in context	fit the chunk (a fraction of the window; leave room for prompt + output); fan out under a concurrency cap
APIs	`GET /documents` returns all	`GET /chunks?cursor=&limit=` with a server-enforced max
Cross-service reads	one request per child	one batched call, capped id lists, partial responses
UI	render the whole list	virtualized / paged; selection survives paging
Stays document-scoped	—	ownership, permissions, billing, ordering / manifest, the unit the user names

You don't have to build every work-layer cell on day one. You have to make sure none of them forbids chunks later — which, concretely, means the chunk exists as a row with its own id from the start. That single seam is the cheap part. Everything else you can defer.

A minimal chunk-native sketch

Almost every chunk-native pipeline I've seen converges on split → map → reduce, with a content-addressed cache guarding the expensive middle:

   split (a tested component)     map: process each chunk          reduce
                                  (memoized + cached)          (the hard part)
   ┌──────────┐  ┌──┬──┬──┬──┐   ┌──────────────────────────┐  ┌───────────────┐
   │ document │─>│c1│c2│c3│c4│─> │ key = hash(chunk, op,    │─>│ dedup + merge │
   └──────────┘  └──┴──┴──┴──┘   │            model, params)│  │  -> glossary, │
                  parallel,      │  hit  -> reuse (0 calls) │  │     graph,    │
                  resumable      │  miss -> call LLM, cache │  │     summary   │
                                 └──────────────────────────┘  └───────────────┘
                       crash at chunk 3?  resume at 3, not at 1.

Three things to notice. split is a component, not a line — with tests, a boundary policy, and a token budget; treating it as first-class is what lets every feature share one notion of "where a chunk begins," and it's the seam that lets you change the grain later without touching storage. reduce is where the hard part moves — combining per-chunk results (dedup entities, stitch a glossary, merge a graph). For map-heavy tasks that's a better problem to have. For tasks whose value is global coherence, one flat reduce won't do it — the fold has to become recursive.

And map needs a governor, or decomposition just relocates the outage. This is the part I'd most want back. Splitting 4,232 chapters into scenes and handing them all to a runtime that will happily start them is not a fix — it's a way of converting one oversized request into thousands of simultaneous ones, which is how you find your provider's rate limits, your connection pool ceiling, and your retry storm all in the same minute. The pattern isn't chunk and parallelize; it's:

bounded decomposition + bounded concurrency + checkpoint + memoization + reconciliation

Every one of those five is load-bearing. Drop the concurrency bound and a successful split becomes a self-inflicted denial of service — worse than the original failure, because now it takes the rest of the system down with it. Cap in-flight work, apply backpressure at the queue rather than the call site, and make retries budgeted rather than automatic.

Three non-obvious tricks that did the real work, none of which the sketch shows:

Anchor injection. Force a handful of globally-critical entities into every window so sparse-but-important facts survive the map phase. (A manual, lossy substitute for real global reduce — see below.)
Position-based id reuse. Match re-parsed pieces on (parent, position) and keep the old id when the hash is unchanged, so an in-place edit doesn't cascade new ids downstream. (Structural edits still cascade — see the identity caveat above.)
Content-hash sweeper. A background pass that re-derives only chunks whose hash changed against the last-processed version — a hand-rolled materialized-view refresh.

Choosing the grain: a four-bound decision procedure

The reason I didn't design chunk-first wasn't a lack of exhortation — it's that at design time you often don't know the right grain, and choosing wrong is itself costly (over-chunking manufactured its own dedup problem — see the appendix). Here's the procedure I wish I'd had. Pick the coarsest unit that satisfies all four bounds — you can always split finer behind split, but you can't cheaply merge:

Operation bound (floor). What's the smallest span one operation must see together to be correct? A scene must stay whole for coreference; a lone sentence can't. Sets the minimum.
Fit bound (ceiling). What's the largest span that comfortably fits one context window / request timeout / transaction — at your P99 real input, not your demo? Sets the maximum.
Identity bound. What's the smallest span a downstream artifact needs to point at and have survive an edit? If nothing references sub-document spans, you may not need chunk identity yet.
Change bound. What's the smallest span that changes independently when the source is edited? That's what your freshness logic wants.

If the bounds disagree, take the coarsest unit that respects the floor, and keep split first-class so you can lower the boundary later. The decision you must get right on day one is not the grain — it's that a sub-document unit with its own identity exists at all. The grain is tunable; the child table is not. (I picked too coarse a grain and still came out ahead, because the seam existed — the full story is in the appendix.)

When it applies, and how to retrofit

When you don't need it

Chunk-first is a cheap early decision, but the elaborate machinery — checkpointing, sweepers, bounded fan-out — is genuinely YAGNI until:

Inputs are unbounded or user-supplied at unknown size. A fixed three-page PDF template never needs it. "Whatever novel the user uploads" always does.
The largest realistic input exceeds one comfortable unit of work — one window, one timeout, one transaction, one screen.
Reprocessing is expensive (LLM calls, embeddings), so caching pays for itself.
The source changes over time and you must re-derive only what moved.

If none hold, the document is your unit and forcing chunks is over-engineering. Even then, the one-line seam (give the unit an id) is cheap insurance — but the stack around it is not, and building it early is the premature-optimization trap.

Coherence-dominated tasks: the reduce changes shape

The tempting thing to say here is that chunking is the wrong architecture when the product is global — "summarize the whole book's theme," "find the plot hole spanning chapters 1 and 4,000," "is this contract self-consistent." That's wrong, and it took me a while to see why.

If the input doesn't fit, decomposition isn't a choice. There is no version of "summarize a 4,000-chapter book" that skips splitting it up — you cannot hold it in one call, so the only question is what you do with the pieces. Coherence-dominated tasks don't make chunk-native the wrong architecture. They make flat, single-pass reduce the wrong reduce.

What changes is the shape of the fold. A flat reduce concatenates per-chunk answers and loses every fact no single chunk ever saw. A recursive reduce compresses in passes: summarize the chunks, group the summaries, summarize those, and repeat until the whole thing fits in one call. It's forging a blade — you fold the steel, and fold the folded steel, and each pass is a genuine compression rather than a concatenation.

That's not a metaphor I invented to feel better about map-reduce; it's precisely what the good systems do. RAPTOR (Sarthi et al. 2024) recursively embeds, clusters, and summarizes chunks into a tree of increasing abstraction. GraphRAG (Edge et al. 2024) builds an entity graph, detects communities, pre-generates a summary per community, then answers by generating partial responses per community and summarizing those into a final answer. Both are chunk-native. Both replace the flat reduce with a hierarchy.

Two honest consequences. Every fold is lossy — you are choosing what survives compression at each level, and that choice is a design decision you should make deliberately rather than discover in an eval. And the "force critical entities into every window" hack in the appendix is what a missing hierarchy looks like when you patch it by hand: I was manually preserving across the fold what a tree reduce would have carried structurally.

So if you're building knowledge-graph-or-summary-over-a-corpus: don't skip decomposition, and don't stop at one reduce.

When it's the wrong architecture

Distinct from both "you don't need it yet" and "your reduce needs another shape": there are tasks where chunk-native genuinely is the wrong architecture even at scale.

Transactional atomicity. "Process this document" as N chunk-jobs can partially fail, leaving a half-finished document — a failure mode document-native never had. Worth separating two things I ran together: chunking the processing does not force you to chunk the commit boundary. You can map over chunks independently and still stage the results and commit the business outcome atomically at the end. What's genuinely a downgrade is making the chunk the transaction boundary — and for financial, medical, or legal documents, where a partial result is worse than none, that's the one to avoid.
Latency-sensitive single-doc ops. Fanning one small document through a queue adds scheduling latency. For "do this one small thing now," a single synchronous call wins.
The window keeps growing. Larger context windows steadily reduce the pressure to chunk for capacity — but they don't touch the operational reasons (checkpointing, memoization, bounded concurrency, incremental re-derivation), and they don't help with coherence across chunks, which is permanent. Expect the capacity argument to weaken over time and the workflow argument not to. Factor that into anything you build today.

How to tell you built document-native

Signs you assumed the whole thing fits — findable in a design review, not a 2 a.m. incident:

Your schema has one row per uploaded document and no child table with its own identity. (This is the one that matters most — it's the seam.)
A core operation says "for every X in the whole document" and runs synchronously in a request.
A job = a document, no checkpoint, so a crash at 90% redoes 100%.
Your list endpoints return everything — no cursor, no page, no cap.
A cross-service read is one request per child (the N+1 query, twenty years old).
Your cache key is the document, so any edit reprocesses all of it.
Your demo dataset is tiny and nobody's run the monster. Test with a monster early — the small input hides every one of these.

The retrofit playbook

If you've already shipped a document-native system — and if you're reading this after hitting a wall, you probably have — the door "decide it first" has closed. Good news: I retrofitted a dozen tables and four subsystems in place, without stopping feature work, and the order matters. Do it in this sequence to keep the blast radius small:

Introduce the seam first, migrate nothing. Add the split component and a chunk table alongside the document table. New writes populate both; reads still use the document. No behavior change, fully reversible.
Backfill bounded, never inline. Migrate existing documents into chunks with a range-scoped, capped batch job — because your backfill is the first unbounded whole-dataset operation you'll write, and it will bite you (it bit me).
Add idempotency and a concurrency cap before parallelism. Put the hash(content, op, model, params) cache key in first, so re-runs during migration are free and safe, and bound in-flight work before you fan anything out. Parallelism without memoization multiplies your blast radius; parallelism without backpressure aims it at your own dependencies.
Cut over reads per layer, cheapest blast radius first: cache → jobs → cross-service reads → API pagination (client-breaking; do it last, behind a version) → UI paging.
Budget the reduce debt explicitly. Per-chunk processing creates a dedup / merge workload that didn't exist before. Make it a line item, not a surprise.
Document the scars. Placeholder ids, versioned columns, deferred fine-splits — record them as known debt with a trigger condition, not silent TODOs.

What it actually cost me

To be square about it, the retrofit wasn't free:

The finest split stayed on my backlog. I split processing to a coarse grain first, not the finest; the cache used a stand-in id in the meantime. Retrofitting is incremental — and half-done chunking has its own sharp edges.
The reduce step is real work. Per-chunk processing created a dedup and bloat problem (thousands of duplicate entities from a handful of chapters) that whole-document processing never had. Chunk-native trades a capability wall for a correctness workload — the right trade, but a trade.
Isolated chunks lose context, and my fixes were ad-hoc. Splitting a book into scenes means each scene is embedded and extracted without the story around it. The cheapest mitigation is the oldest — chunk overlap / stride — which I under-used. For retrieval specifically, contextual retrieval (Anthropic, 2024: prepend a short generated context blurb before embedding) beats anything I hand-rolled. Anthropic reports it cut the top-20-chunk retrieval failure rate by 35% (5.7% → 3.7%), by 49% combined with contextual BM25, and by 67% with reranking added. Those are their own internal evaluations rather than a peer-reviewed result — but the gap is wide enough that I should have started there instead of inventing my own mitigations. (Note: late chunking is a retrieval-embedding technique — it would not have helped my extraction context loss; I'd conflated them.)

I'm not claiming chunk-first is free. I'm claiming the seam is nearly free, and the retrofit without it is what you sign up for by default.

None of this is new

I invented none of these primitives, and the argument is stronger for admitting it. Every one has an older name:

Map-reduce over chunks (Dean & Ghemawat, 2004) — the standard pattern for work that exceeds one machine or one context. The "stuffing vs. map-reduce vs. refine" taxonomy is textbook.
Declare the grain (Kimball) and records-not-files / partitioning (Kleppmann, Designing Data-Intensive Applications) — the data-modeling half, decades old. DDIA is the single best reference for this entire post.
Semantic operators — map / filter / reduce as first-class — DocETL and the emerging "semantic operators" research line for LLM data pipelines. My "the caller handles chunking" contract boundary is exactly that.
Content-addressed caching / durable execution (Temporal, Restate, LangGraph) — checkpoint, replay, resume. My "checkpoint + parallel map + hash key" is durable execution, reinvented with worse tooling.
Incremental View Maintenance, CDC, differential dataflow — the mutable-source freshness problem, solved for decades in data engineering.

So why write it? Because those names live in two communities that don't talk. The RAG world owns the word "chunking" and means retrieval. The data-engineering and durable-execution world owns "grain," "decomposition," and "idempotency" and means the unit of work. The failure I lived was the gap between them: I'd read plenty about retrieval chunking, saw the word everywhere, and concluded I understood chunking. I understood one layer. The contribution isn't a new primitive — it's insisting the same grain decision has to be made once, early, and honored (as scale demands) at every work layer, instead of re-litigated bug-by-bug.

One more correction to a line I've seen everywhere (and wrote myself): you don't chunk because "attention is O(n²)." The quadratic term is real — it's still quadratic FLOPs, and you pay it on every prefill — but it isn't the binding constraint. FlashAttention made it memory-linear (the n×n matrix is never materialized), sparse and linear variants attack the compute directly, and KV-cache memory is O(n) regardless. None of that is what stops you. You chunk because of a hard context cap, quality that can decay well before it (lost in the middle, Liu et al. 2023), token cost, and request timeouts. Every one of those is an engineering limit, not a complexity class — which is exactly why the fix lives in your architecture and not in the model.

Closing

Every one of these failures wears an LLM costume. The feature "fails" at the model; the pipeline "fails" at the model session. It's tempting each time to file the bug as "make the prompt smaller." Sometimes the prompt really is the problem — bad instructions and irrelevant context are their own bugs. But when the trigger is size, the prompt is only where the problem first became visible. The problem is the layers beneath it all quietly agreeing that a document is a unit — and a large document isn't a unit. It's a container of many units the system keeps mistaking for one.

And in my case I couldn't even plead ignorance — I'd written "any size, up to 50 MB+" into the goals myself. The gap was never knowledge; it was the discipline to let that knowledge reach the schema. That's why the fix is a checklist, not a fact: the fact was already in my own requirements.

So decide the grain early, and keep the seam open:

Chunk-first (the cheap part): one row with its own id, from day one.
Chunk-native (the deferred part): as scale arrives, push the chunk through jobs, caching, LLM calls, and APIs — bounded decomposition, bounded concurrency, checkpoint, memoization, reconciliation — while ownership and meaning stay with the document.

Stated precisely: for unbounded or mutable inputs, the processing grain is a data-model decision, not a prompt-time optimization. Or, less carefully but more usefully: chunking is not something you do to a prompt. It's the grain of the system. Decide it early — not because retrofitting is impossible, but because every month you wait adds data to migrate and callers to update. It's the same decision either way. Only the price changes.

Appendix — the lore-weave case study, in depth

Everything above is the general lesson. This is where it came from — the real system, the real quotes, the real numbers. Read it if you want the receipts; skip it if the principle was enough.

Lore-weave (source on GitHub) is an open-source platform for building a fictional world and then playing it. Authors and AI agents jointly maintain the knowledge of a long-running novel — it renders a source into the reader's language, extracts a glossary of characters, places, and items, and builds a knowledge graph of who did what to whom across the story — and that accumulated knowledge is the substrate an RPG world simulator is designed to run on. Everything in this post concerns the knowledge spine, which is where all four failures happened. Its stated goal, written before any code, was to "handle novels of any size, up to 50 MB+, on a local-first stack."

The requirement was right there in the goals — and the design assumed the opposite of it anyway. The book that exposed the gap was a real 4,232-chapter web-novel; almost every break below was found against that one book. The four failure modes from the top of the post, in the order I actually hit them:

Failure 1 — Translation assumed a chapter fits in one prompt

Trap 1 (the unbounded unit) in practice. Translation ran one chapter at a time, sent whole to the model as a single prompt — in the redesign doc's words:

"The current translation pipeline sends an entire chapter as a single prompt to the model. This breaks in two ways. Context overflow — the chapter text is small on its own, but translating it consistently means injecting the book's ever-growing glossary and prior context alongside it; on a large book that combined payload blows past even a 100K+ context window, so the model silently truncates or refuses. Timeout cascade — one giant prompt takes too long, hits the request timeout, and the whole chapter fails."

Notice it broke before "the whole book" was even the problem. Even "one chapter" wasn't a bounded unit: the context it needs to translate consistently — the glossary — grows with the book, so the per-chapter payload creeps upward the deeper you read. The fix introduced a splitter (break on sentence and paragraph boundaries, up to a token budget) and a translation session that carries state from chunk to chunk and periodically compacts its own history to stay inside the window.

But fixing the prompt only revealed the next layer. The translation UI showed one row per translated slice — and at thousands of chapters that surface needed its own rework: pagination, a selection model that survives paging, and an abortable, paged loading loop, because merely fetching the list of chapters is now a batched operation. The prompt was one chunk-boundary; the chapter list was another. Different layers, discovered months apart.

Failure 2 — Glossary extraction did O(n) scans and one runaway backfill

Trap 2 (the whole-dataset operation) in practice. It showed up twice. First, the glossary — the running list of characters, places, and items the system has seen — is built by reading each chapter and checking every candidate name against everything already stored. That check was a linear scan over every stored entry: O(n), and "at 10,000 entries that's 10,000 scans per extraction." (Narrowly, the fix here is "add an index" — and a skeptic is right to say so. The point is why the index became load-bearing: the glossary grows with the number of chunks, so on a 6-chapter test novel you never feel it, and on a 4,000-chapter one it's a quadratic wall.)

The sharper example was a runaway. Turning on the embeddings feature kicked off a one-time job to go back and process every existing chapter — with no cap on how much it would do in a single request:

"Setting the project's embedding model fires a synchronous, in-request backfill over every published chapter of the book — with no scope limit. On the 4,232-chapter book it embedded ~11,600 passages before a manual restart stopped it."

The fix made the operation bounded by construction: a chapter-range parameter plus a hard inline cap (200 chapters by default). The logic wasn't wrong — it was whole-book logic in a place that should only ever touch a bounded slice.

Failure 3 — Knowledge-graph building treated a chapter as one atomic job

Trap 3 (the all-or-nothing job) in practice. Building the knowledge graph means having the model read the book and pull out who-did-what — and the original design made each chapter one such job. Its original sin, stated flatly:

"The current pipeline cannot scale because every chapter is treated as one extraction job, with serial chunks inside it and flat, key-only dedup across them. Sessions longer than ~1 hour reliably evict the model from memory … and there is no checkpoint or resume."

The reframe that fixed it:

"Scale is wall-clock, not capability. With checkpoint + parallel map + an idempotent task id, a local mid-size model is fully capable of any size."

That's true for work that decomposes cleanly — per-scene translation, per-scene entity extraction. Tasks with genuine long-range dependencies (global coreference, cross-chapter contradiction, "what is the theme") still decompose — they just need a recursive reduce rather than a flat one. For the map-heavy 80%, a single fold holds. The redesigned engine says so in its own header:

"What this module deliberately does NOT do: chunking — the caller handles splitting. Cost tracking — the caller manages the budget."

Chunking became a contract boundary between components, not a step buried in one function. The unit of extraction became a scene (a sub-chapter passage), not the whole chapter. Each extraction step is keyed by a content hash of (chunk text, operation, model version, output schema), so re-running an unchanged chapter is all cache hits and zero model calls. And because the graph spans the whole book, it needs a trick that betrays the limits of pure map: a handful of critical names are force-injected into every window, so a character who appears in chapter 1 and again in chapter 4,000 stays anchored — a hand-patched substitute for the global reduce that flat map-reduce can't do.

Failure 4 — The container was thrown away

Trap 4 (the discarded structure) in practice — and the odd one out. The first three failures are all about the atom: the unit of work was too big, unbounded, or un-checkpointed. This one is about the container, and it's worth being precise that it is a different kind of bug. Discarding the source of a derivation isn't a size problem — you can do it with six chapters and a perfect chunk model. What size changes is whether you can recover: at six chapters you re-derive the outline by reading everything, and at 4,232 you cannot. Scale doesn't cause this failure. It makes it permanent.

It belongs here anyway, because it's the half of chunk-native the other three don't touch. Once the document stops being the unit of work, it has to become something else — a durable container carrying the ordering, the manifest, the parent ids. That's the thing that makes chunks reassemblable into a book. Throw it away and a decomposition degrades into an unordered bag of parts.

The system first plans a book's high-level structure — its outline and story arcs — then generates the real chapters from that plan. Once the chapters existed, the plan was thrown away. My own complaint, the one that kicked off the refactor (lightly cleaned up from the original):

"After we finish planning and generate the real book, we only keep the chapters and lose the architecture of the book. It's like you compile source code and then throw the source away — you keep only the binary."

The refactor re-scoped a dozen tables so the book — not the user's project — became the primary key, with batched backfills sized for 10,000-chapter books, and a browser query that fetched arcs (the outline units that group chapters) one request at a time — O(arcs), not O(chapters) — collapsed into a single call. The index that maps structure back onto prose became a content-hash-preserving, per-chapter upsert with a background sweeper that re-processes only the chapters whose text actually changed — the same move every build system makes: keep the structure as durable, diff-able data, not a byproduct you discard once the output exists.

The pattern: four bugs, one assumption underneath

Feature	The "self-sufficient document" assumption	Half	The layer it broke at	The bug's usual name
Translation	a chapter is one prompt	it fits	model call, then the UI list	context overflow + no pagination
Glossary	scan / backfill the book inline	it fits	database + request timeout	missing index + unbounded request
Knowledge graph	a chapter is one atomic job	it fits	job runner + model session	no checkpoint / resume
Book structure	the chapters carry the book's structure	it describes itself	schema + cross-service queries	wrong scope key + stale index

Four separately-named bugs, one shared assumption underneath — three of them the atom half, one the container half. None is a model problem; every one only surfaced at the model. That's the trap: chunking looks like an LLM concern because that's where the first symptom appears. It's a system concern, and the LLM call is the last place you find out.

Prior art and further reading

The value here is unification, not novelty. The pieces, in their native fields:

Declare the grain / data modeling — Ralph Kimball, The Data Warehouse Toolkit (the four-step design process; "declare the grain" is step 2) · Martin Kleppmann, Designing Data-Intensive Applications (the canonical modern reference for records, partitioning, batch vs. stream).
Decomposition / map-reduce — Dean & Ghemawat, "MapReduce: Simplified Data Processing on Large Clusters" (OSDI 2004) · DocETL — Shankar et al., UC Berkeley EPIC Lab (arXiv:2410.12189), which exposes map / reduce / filter as first-class operators over documents · Semantic operators / LOTUS — Patel et al. (arXiv:2407.11418), the declarative model that names them.
Global sensemaking over a corpus (the reduce-is-lossy answer) — GraphRAG, Edge et al., "From Local to Global: A Graph RAG Approach to Query-Focused Summarization" (arXiv:2404.16130) · RAPTOR, Sarthi et al., "Recursive Abstractive Processing for Tree-Organized Retrieval" (arXiv:2401.18059).
Durable execution — What is durable execution? · LangGraph durable execution.
Mutable-source freshness — Incremental View Maintenance; Change Data Capture; Differential Dataflow (McSherry, Murray, Isaacs & Isard, CIDR 2013); self-adjusting computation (Umut Acar).
Retrieval-chunking refinements (the other meaning of the word) — Anthropic, "Introducing Contextual Retrieval" (2024) · Late Chunking — Günther et al. (arXiv:2409.04701) · Weaviate · Unstructured.

This is episode #02 of the AI Engineering series. Episode #01 — When an AI Discards Its Own Search Results is about a different failure mode (belief retention), but shares this one's spine: the hard part of an AI system is rarely the model call — it's the architecture around it. The case study is a real system I built — lore-weave, open source — and every quoted line is my own, from that repo's design docs, written during a large-scale refactor. You can go read them in context.

DEV Community