DEV Community

Tejas Kadam
Tejas Kadam

Posted on

chunk-lite: token-aware text chunking for RAG pipelines that doesn't cut mid-sentence

#ai #typescript #webdev #opensource

Package #5 in my small open-source series, and the first one with a direct RAG/AI angle.

Every RAG pipeline starts the same way: take a long document, split it into pieces small enough to embed, and hope the split points don't wreck the meaning inside each piece. A naive fixed-size split has no idea where a sentence or paragraph actually ends, so it happily cuts a chunk in half mid-thought - and a chunk that starts "...and that's why the deadline was moved" with no antecedent for "that" embeds as a blurry average of two unrelated ideas. It retrieves worse for both.

chunk-lite makes the boundary-aware version the default.

import { chunkText } from 'chunk-lite';

const chunks = chunkText(longDocument, {
  maxTokens: 300,
  overlapTokens: 50,
});

chunks[0];
// { text: "...", startOffset: 0, endOffset: 1180, tokenCount: 298, index: 0 }
Enter fullscreen mode Exit fullscreen mode

A few design notes:

  • Chunks are packed sentence-by-sentence by default (falling back to word boundaries only when a single sentence alone exceeds maxTokens), so every chunk is a coherent unit of text, not a fragment.
  • Overlap is configurable, not automatic - it prevents a fact from being invisible to retrieval just because it landed on a chunk boundary, but it also multiplies embedding calls and storage, so there's no universally "correct" default.
  • Every chunk carries startOffset/endOffset into the source text, so you can always trace a retrieved chunk back to exactly where it came from - for citations, highlighting, or re-chunking later without losing the mapping.
  • No hard dependency on any specific tokenizer. The default counter is a dependency-free ~4-chars-per-token heuristic; pass your own tokenCounter (tiktoken, a model's own endpoint, whatever) if you need exact counts.

Fully typed, zero required runtime dependencies, 26 tests covering overlap correctness, sentence/paragraph boundaries, oversized-sentence fallback, and custom tokenizer injection.

GitHub: https://github.com/tejas821/chunk-lite
npm: npm i chunk-lite

Full reasoning on chunk boundaries and the overlap trade-off is in the repo's CASE_STUDY.md. Feedback welcome, especially if you've hit a boundary case this doesn't handle well.

Top comments (0)