DEV Community

Solon Framework
Solon Framework

Posted on

Chunk by Meaning, Not Just Size: A Deep Dive into Solon AI's SemanticSplitter

#ai #java #opensource #rag

Chunk by Meaning, Not Just Size: A Deep Dive into Solon AI’s SemanticSplitter

RAG quality is often discussed as if the only question were which vector database to choose. In practice, the shape of the text entering that database matters just as much. If one chunk mixes a refund rule, a shipping exception, and an account-security note, a strong embedding model cannot fully repair the damaged boundary.

Solon AI’s SemanticSplitter offers a different trade-off from regex- and token-based splitting: it uses an embedding model to look for likely topic boundaries before applying a token-size fallback.

This article examines what the implementation actually does in Solon AI v4.1.0, where it fits in the RAG pipeline, and which assumptions an application should still validate.

The splitter is an ingestion-stage component

Solon’s RAG documentation separates the main stages clearly:

DocumentLoader -> DocumentSplitter -> RepositoryStorable.save(...)
Enter fullscreen mode Exit fullscreen mode

DocumentLoader turns a file or other source into Document objects. A DocumentSplitter can then turn large documents into smaller retrieval units. The resulting documents are finally stored in a Repository.

SemanticSplitter belongs to the middle stage. It does not query a vector database and it does not decide which documents to retrieve at runtime. Its job is to decide where the boundaries of the documents should be before indexing.

The class is:

org.noear.solon.ai.rag.splitter.SemanticSplitter
Enter fullscreen mode Exit fullscreen mode

and it implements the same DocumentSplitter abstraction used by other Solon AI splitters.

What the default constructor configures

The simplest construction is:

SemanticSplitter splitter = new SemanticSplitter(embeddingModel);
Enter fullscreen mode Exit fullscreen mode

The source defines these defaults:

Setting Default Meaning
similarityThreshold 0.5 A lower similarity marks a candidate boundary
maxChunkTokenSize 512 Target maximum size for a semantic group
similarityWindow 3 Number of preceding sentences used as context
minSentencesPerChunk 1 Minimum distance between accepted boundaries
delimiters ALL_COMMON_DELIM Common English, Chinese, and newline delimiters

These are defaults, not universal production settings. Similarity distributions vary between embedding models, languages, and corpora. A threshold that is useful for one model can be too aggressive or too conservative for another.

The algorithm is a sliding-window comparison

The implementation does not embed the entire document once and then run a generic clustering algorithm. Its flow is more specific:

raw text
  -> sentence-like segments
  -> sliding context windows
  -> embeddings for windows and following sentences
  -> cosine similarities
  -> threshold-based boundaries
  -> token-size fallback
  -> new Documents with inherited metadata
Enter fullscreen mode Exit fullscreen mode

1. Sentence-like segmentation

The splitter scans the text using the configured delimiters. The delimiter remains attached to the preceding segment. The built-in common set includes:

SemanticSplitter.ALL_COMMON_DELIM
Enter fullscreen mode Exit fullscreen mode

which covers ., !, ?, Chinese 。, ！, ？, ；, and newlines.

This is deliberately simpler than a full natural-language sentence parser. If a document uses unusual punctuation or has long sections without the configured delimiters, the resulting segments will reflect that.

2. Build a context window

For each position, the splitter concatenates the next similarityWindow sentences into a window. With the default window of three, the comparison looks conceptually like this:

[sentence 0 + sentence 1 + sentence 2]  vs  [sentence 3]
[sentence 1 + sentence 2 + sentence 3]  vs  [sentence 4]
[sentence 2 + sentence 3 + sentence 4]  vs  [sentence 5]
Enter fullscreen mode Exit fullscreen mode

Each side is embedded. The cosine similarity between the window and the following sentence becomes the signal for that position.

A low score suggests that the next sentence may represent a topic transition. It is a candidate boundary, not a semantic truth guarantee.

3. Apply the threshold

The source uses the direction below:

if (similarities[i] < similarityThreshold) {
    // candidate split
}
Enter fullscreen mode Exit fullscreen mode

This direction matters. Holding the text, model, and other parameters constant, increasing the threshold makes it easier for a position to fall below the threshold, so it will usually create more candidate boundaries. Lowering the threshold usually keeps larger groups together.

The result is not a simple linear knob. The final chunks also depend on the embedding model, the window size, the minimum-sentence rule, and the token fallback.

4. Enforce a token-size fallback

After semantic groups are created, the splitter counts tokens using its configured jtokkit encoding. If a group is over maxChunkTokenSize, it allocates sentences into smaller groups until the group would exceed the limit.

That makes the token limit a second-stage guardrail, not the primary boundary detector.

There is an important edge case: the fallback only moves whole sentences. If one individual sentence is already larger than the configured maximum, it can still appear as an oversized output chunk. Also, very short documents with no more than similarityWindow segments are returned as one document before the normal embedding and token fallback path runs.

So the safe statement is:

SemanticSplitter applies a token-size fallback to oversized semantic groups, but applications should still validate final chunk sizes for short documents and individually oversized sentences.

Embedding cost is part of the design

Regex splitting and token splitting can be performed locally. Semantic splitting calls the configured EmbeddingModel.

For a document with N segments and a window size of W, the normal path builds approximately:

N - W window texts
N - W following-sentence texts
2 * (N - W) embedding inputs
Enter fullscreen mode Exit fullscreen mode

The inputs are sent in batches according to embeddingModel.batchSize(). The current implementation also requests:

.options(o -> o.dimensions(512))
Enter fullscreen mode Exit fullscreen mode

That detail deserves operational attention. The selected embedding provider must support the requested dimension behavior, and the application should measure whether the chosen dimension preserves acceptable boundary quality for its corpus. Do not assume that every provider interprets dimensionality options identically.

The practical cost model is therefore different:

Splitter Boundary signal Typical cost Main risk
RegexTextSplitter Document structure Local CPU Structure may be inconsistent
TokenSizeTextSplitter Token count Local CPU A chunk can cross topic boundaries
SemanticSplitter Embedding similarity Embedding calls plus CPU Cost and thresholds need validation

This is not a ranking in which the semantic splitter always wins. It is a choice between different failure modes.

Metadata is preserved, but chunk identity is not invented

When the splitter creates a new Document, it passes the original metadata through:

new Document(content, metadata)
Enter fullscreen mode Exit fullscreen mode

That is useful for filters such as:

  • source file;
  • department;
  • tenant;
  • document version;
  • access scope.

However, the splitter does not automatically add a chunk number, parent-document ID, page number, character offsets, or neighboring-chunk links. If those fields are needed for citation, deletion, re-indexing, or audit trails, add them in the application’s ingestion layer rather than assuming they are framework-generated.

A production ingestion pipeline might therefore enrich the metadata after splitting:

List<Document> chunks = splitter.split(loadedDocuments);

for (int i = 0; i < chunks.size(); i++) {
    // Add application-owned tracking fields here:
    // chunkIndex, sourceId, pageNumber, version, and so on.
}

repository.save(chunks);
Enter fullscreen mode Exit fullscreen mode

The exact metadata API depends on the application’s ownership model. The important design point is that chunk tracking is an application responsibility unless the loader already supplies the required fields.

How to evaluate it instead of guessing

Semantic chunking should be evaluated with the same discipline as a retrieval model. A useful experiment can use one mixed-topic corpus containing, for example:

  1. refund rules;
  2. shipping compensation;
  3. account-security requirements;
  4. similar vocabulary across all three sections.

Compare at least these measurements:

  • number of produced chunks;
  • average and P95 token count;
  • number of chunks crossing a known section boundary;
  • embedding request count;
  • ingestion latency;
  • Recall@K on a fixed question set;
  • retrieved-context token count;
  • whether the final answer cites the correct section.

A small parameter sweep is also more informative than copying defaults blindly:

similarityThreshold: 0.35 / 0.50 / 0.65
similarityWindow:    1 / 3 / 5
maxChunkTokenSize:   256 / 512 / 1024
Enter fullscreen mode Exit fullscreen mode

Report results with the corpus, embedding model, parameter values, and measurement method. Without those details, a claim such as “semantic splitting improves accuracy” is too broad to be reliable.

Where Agent RAG fits

Solon AI has separate concepts for ingestion and runtime retrieval. A useful mental model is:

SemanticSplitter
      -> better-shaped Documents
      -> Repository
      -> RepositoryTool
      -> ReActAgent
Enter fullscreen mode Exit fullscreen mode

These components answer different questions:

  • SemanticSplitter: where should the knowledge be divided?
  • Repository: where are the indexed documents stored and searched?
  • RepositoryTool: how can retrieval be exposed as a tool?
  • ReActAgent: when should the agent search, and whether it should search again?

Improving chunk boundaries can help the retrieval layer, but it does not turn the splitter into an agent. Conversely, an agent cannot reliably compensate for every bad ingestion boundary. Treat the ingestion strategy and runtime strategy as separate, measurable layers.

A practical decision guide

Use a structure-based splitter when:

  • headings and paragraphs are reliable;
  • indexing cost must be minimal;
  • deterministic boundaries are important;
  • you need straightforward debugging.

Use a token splitter when:

  • model input limits are the dominant constraint;
  • the corpus has weak structure;
  • you want predictable chunk sizes;
  • the retrieval quality trade-off is acceptable.

Evaluate SemanticSplitter when:

  • the corpus contains frequent topic transitions inside long sections;
  • structural delimiters are not enough;
  • you can afford embedding work during ingestion;
  • you have a representative retrieval benchmark.

A hybrid pipeline can also be reasonable, but its order and benefit should be tested on the target corpus rather than declared as a universal best practice.

Closing thoughts

SemanticSplitter is interesting precisely because it is not magic. It is a concrete algorithm with visible costs and boundaries:

  • sentence-like segmentation first;
  • sliding-window embeddings;
  • cosine similarity as a boundary signal;
  • threshold-based grouping;
  • token-size fallback;
  • metadata inheritance;
  • no automatic chunk identity model.

That makes it possible to reason about, benchmark, and adapt. In a Solon AI RAG pipeline, the best splitter is not the one with the most sophisticated name. It is the one whose boundary behavior, embedding cost, and retrieval results are understood on your own documents.

Sources

Top comments (0)