DEV Community: Speedyk-005

Don't delete your old code. It might be someone's next starting point.

Speedyk-005 — Wed, 29 Jul 2026 05:52:37 +0000

Every developer has code they'll probably never touch again.

The half-finished CLI from a weekend.
The tutorial project that stopped halfway through.
That weird experiment that actually taught you something.
The utility script that's ugly but gets the job done.

Most of that code just sits on a hard drive until it's forgotten.

I started The Open CodeYard, a GitHub repository for unfinished, experimental, and abandoned projects that are still worth preserving. Not because they're perfect, but because ideas are often more valuable than polished code.

The structure is simple:

"scraps/" for random leftovers.
"prototypes/" for intentional experiments.

Each project gets a small README explaining what it does and what state it's in.

A few ground rules:

No clean-code police.
Messy code is expected.
Abandoned projects are welcome.
If you're thinking, "my code isn't good enough," you're exactly who this repo is for.

Right now it's just my own projects in there, including a few C# pathfinding algorithms, a speech rate estimator, and some Python utilities. I'm hoping it grows into a community archive where forgotten code can still teach, inspire, or even get revived by someone else.

If you've got code collecting digital dust, I'd love to see it rescued instead.

Repository:
https://github.com/speedyk-005/TheOpenCodeYard

Replacing spaCy’s Sentencizer with yasbd: From 55.4% to 98.9% Accuracy

Speedyk-005 — Mon, 27 Jul 2026 23:05:04 +0000

Sentence segmentation—splitting text into individual sentences—is a foundational step in most NLP pipelines. It seems simple, but as anyone who has worked with real-world text knows, the details matter. Abbreviations like Dr., URLs, ellipsis, and complex punctuation can trip up even well-intentioned rule-based systems.

spaCy's built-in Sentencizer is a fast, lightweight, rule-based component designed for this task. However, by design, it makes a trade-off: speed and simplicity over accuracy on edge cases. For many production applications, this trade-off leads to catastrophic downstream errors.

Enter yasbd (Yet Another Sentence Boundary Detector). It's a pure-Python, rule-based SBD that can be integrated as a drop-in spaCy component, and it achieves a remarkable 98.9% accuracy on a challenging English golden benchmark, compared to spaCy's Sentencizer which scores only 55.4% [1].

This post will walk you through why spaCy's default segmenter falls short, and how you can easily upgrade your pipeline with yasbd to achieve near-perfect sentence boundaries.

Why spaCy's Sentencizer is Intentionally Lightweight

The Sentencizer is a non-trainable pipeline component. Reading its source code reveals its core logic is only a few dozen lines [2]:

Mark the first token of the document as a sentence start.
Walk through every token in the document.
If a token is in punct_chars (., ?, !, 。, etc.), remember that a sentence may end here.
Skip over any following punctuation.
As soon as the next token is not punctuation, mark it as the beginning of a new sentence.

In simplified pseudocode, it looks like this:

start = 0
seen_period = False
doc_guesses[0] = True

for token in doc:
    if token.text in punct_chars:
        seen_period = True
    elif seen_period:
        doc_guesses[token.i] = True
        seen_period = False

That's it. It performs no abbreviation list lookup, no context analysis, and no handling of quotes, parentheses, or URLs. As the spaCy documentation itself notes, it's a component for when you "don't require the dependency parse" and want a "simpler, rule-based strategy" [3]. It was never designed to be a robust sentence splitter for messy, real-world text.

Wait, Doesn't spaCy Handle Abbreviations?

You might be wondering: "But spaCy handles Dr. just fine, doesn't it?" This is a sharp observation, and it gets to an important distinction in how spaCy processes text.

The Sentencizer itself has no abbreviation awareness, but spaCy's Tokenizer runs first and applies hard-coded rules called Tokenizer Exceptions for common patterns like Dr., Mr., Ms., and U.K. [5, 6].

Here's what happens:

· Dr.: The Tokenizer keeps it as a single token ['Dr.']. The Sentencizer sees this as one unit and often bypasses the boundary check.
· A.: The Tokenizer splits this into 'A', '.'. The Sentencizer now sees a standalone period token and forces a sentence split.
· M.D.: No built-in exception exists, so the Tokenizer splits it into ['M', '.', 'D', '.']. The Sentencizer sees multiple standalone periods and fragments the text.

This is why the Sentencizer works on simple cases like "Dr. Smith" but collapses on compound abbreviations, citations, or text with multiple initials. The tokenizer provides some protection, but it's incomplete and inconsistent [4, 6].

A GitHub issue from 2021 highlights this limitation: a user pointed out that for Polish, the Sentencizer doesn't handle common abbreviations like dr., prof., or inż., making it effectively broken for that language [6]. The response from the spaCy team acknowledged that adding default exceptions for all languages is challenging, as the tokenizer defaults follow the guidelines of specific training corpora [7]. This means that for the vast majority of the 75+ languages it claims to support, the Sentencizer has minimal abbreviation awareness.

The Benchmark Gap: 55.4% vs. 98.9%

The performance difference becomes stark when you test these tools on a diverse set of challenging English texts. The yasbd benchmark suite includes a golden dataset of 92 English edge cases, expanded from the original 48 cases used by pysbd. These cases are specifically designed to break naive splitters and include abbreviation chains, URLs, quotes, ellipsis, and academic citations [1].

The results from the yasbd benchmarks are clear [1]:

Library	Score on 92 Edge Cases
yasbd	91/92 (98.9%)
pysbd	77/92 (83.7%)
sentencex	76/92 (82.6%)
blingfire	75/92 (81.5%)
sentsplit	61/92 (66.3%)
sentence-splitter	60/92 (65.2%)
nupunkt	59/92 (64.1%)
spacy-sentencizer	51/92 (55.4%)

This puts spacy-sentencizer at the very bottom of the list. It fails on nearly half of the test cases, primarily because it has no concept of abbreviations and will blindly split on any period, regardless of context.

How to Install yasbd-lib

The installation is straightforward via pip. Note that yasbd-lib is a pure-Python library with no native dependencies, making it easy to install in any environment.

pip install yasbd-lib -U

That's it.

Integrating yasbd as a spaCy Component

This is where the power of yasbd really shines for spaCy users. It provides a simple one-function registration API, allowing you to replace the default sentencizer with just a few lines of code [8].

spaCy is not a dependency of yasbd. You will need to install it separately.

pip install spacy -U

Once you have both libraries, the integration is seamless. The key is to add the yasbd component first in your pipeline. This ensures its sentence boundaries are set early and can be used by subsequent components like the parser or NER, rather than being overwritten.

import spacy
from yasbd import register_spacy_component

# 1. Register the component. This tells spaCy about the "yasbd" factory.
register_spacy_component()  # Requires spaCy v3+

# 2. Create a blank English pipeline.
nlp = spacy.blank("en")

# 3. Add the yasbd component FIRST.
#    `first=True` ensures it runs before the parser, so its boundaries are preserved.
nlp.add_pipe("yasbd", first=True, config={"lang": "en"})

# 4. Process your text.
doc = nlp("Dr. Smith arrived. He was late.")

# 5. Access the beautifully segmented sentences.
for sent in doc.sents:
    print(sent.text)
# Output:
# Dr. Smith arrived.
# He was late.

Configuration

The yasbd component is highly configurable. You can set the language, enable verbose logging, or even use automatic language detection [8].

# Language can be inherited from the pipeline's language
nlp.add_pipe("yasbd", first=True)  # `lang` defaults to `nlp.lang`

# Or, use auto-detection (slower, but useful for mixed-language text)
nlp.add_pipe("yasbd", first=True, config={"lang": "auto"})

# Or, configure it at runtime
pipe = nlp.get_pipe("yasbd")
pipe.detector.lang = "fr"  # Switch to French rules
pipe.detector.verbose = True
pipe.preserve_quote_and_paren = False  # Disable quote/parenthesis protection

Before and After Examples

Let's see the difference in action on a piece of text designed to break the Sentencizer.

import spacy
from yasbd import register_spacy_component

register_spacy_component()

# --- Pipeline with spaCy's default sentencizer ---
nlp_default = spacy.blank("en")
nlp_default.add_pipe("sentencizer")

# --- Pipeline with yasbd ---
nlp_yasbd = spacy.blank("en")
nlp_yasbd.add_pipe("yasbd", first=True, config={"lang": "en"})

text = """Dr. Patel A. (M.D., Ph.D.), can corroborate my claim.
You can reach me at j.doe@example.com or visit https://www.example.com/page?ref=1.
As Smith et al. (2021) noted: "The implications are far-reaching." However, critics disagree."""

print("--- spaCy Sentencizer ---")
for sent in nlp_default(text).sents:
    print(f"'{sent.text}'")
# Output:
# 'Dr. Patel A. (M.D., Ph.D.), can corroborate my claim.'
# '
# You can reach me at j.doe@example.com or visit https://www.example.com/page?ref=1.'
# '
# As Smith et al. ('
# '2021) noted: "The implications are far-reaching."'
# 'However, critics disagree.'
#
# Notice: Newlines become sentence starts, and "2021" becomes its own fragment.

print("\n--- yasbd ---")
for sent in nlp_yasbd(text).sents:
    print(f"'{sent.text}'")
# Output:
# 'Dr. Patel A. (M.D., Ph.D.), can corroborate my claim.
# '
# 'You can reach me at j.doe@example.com or visit https://www.example.com/page?ref=1.
# '
# 'As Smith et al. (2021) noted: "The implications are far-reaching."'
# 'However, critics disagree.'

The Sentencizer output is a mess:

· It splits on newline characters, creating empty sentence fragments.
· It shatters the citation "(2021)" into its own sentence.
· It completely fails to recognize compound abbreviations like M.D., Ph.D..

yasbd, with its comprehensive two-pass system—first identifying candidate boundaries, then surgically removing false positives using abbreviation lists and context rules—handles the entire text flawlessly [8].

Conclusion

Sentence segmentation is a critical first step in any NLP pipeline. While spaCy's default Sentencizer is a convenient, lightweight option, its simplistic, period-based logic makes it highly inaccurate on real-world text. While the Tokenizer provides some protection for common abbreviations like Dr., this protection is incomplete and fails on compound abbreviations, initials, and non-English text [5, 6, 7]. The benchmarks show it failing on nearly half of the tested edge cases [1].

yasbd offers a compelling alternative. It is a high-accuracy, rule-based SBD that can be integrated as a spaCy component with just a few lines of code, providing a massive leap in accuracy from 55.4% to 98.9% [8].

For any spaCy user who cares about the quality of their sentence boundaries—and the downstream tasks that depend on them—replacing the sentencizer with yasbd is an easy and highly effective upgrade.

References

[1] yasbd-lib benchmarks. https://github.com/speedyk-005/yasbd-lib/blob/main/benchmarks/README.md

[2] spaCy Sentencizer source code. https://github.com/explosion/spaCy/blob/master/spacy/pipeline/sentencizer.pyx

[3] spaCy Sentencizer API documentation. https://spacy.io/api/sentencizer

[4] Stack Overflow: spaCy split sentences with abbreviations. https://stackoverflow.com/questions/53968330/spacy-split-sentences-with-abbreviations

[5] spaCy Linguistic Features documentation. https://spacy.io/usage/linguistic-features

[6] GitHub issue: Tokenizer exceptions for Sentencizer (#7218). https://github.com/explosion/spaCy/issues/7218

[7] Comment by adrianeboyd on issue #7218. https://github.com/explosion/spaCy/issues/7218#issuecomment-786685415

[8] yasbd-lib spaCy component API. https://github.com/speedyk-005/yasbd-lib#spacy-component-api

Four ways to split sentences (without machine learning)

Speedyk-005 — Fri, 24 Jul 2026 03:04:24 +0000

Before a machine can summarize text, translate documents, extract entities, or answer questions, it first needs to know where each sentence begins and ends. Get the boundaries wrong, and downstream tasks inherit corrupted input.

This article explores the design space for rule-based sentence boundary disambiguation (SBD) systems, examining the architectural choices that determine correctness, performance, and maintainability.

The Problem: Punctuation Is Overloaded

The obvious solution seems almost embarrassingly simple:

re.split(r'(?<=[.!?])(?=\s+[A-Z])', text)

It works... until it doesn't. A period alone does six jobs and only one is "sentence end." Here are just a few situations where naive regex fails:

Input	Why it fails
`Dr. Smith arrived.`	Dr. is an abbreviation, not a sentence end
`Apple Inc. announced...`	Inc. isn't the end
`The value is 3.14159.`	Decimal point
`D. H. Lawrence wrote...`	Initials
`Wait... what?`	Ellipsis
`Version 2.5.1 fixes bugs.`	Software version
`Visit example.com.`	Domain name
`1. Install Python.`	Numbered list

Studies estimate that approximately 47% of periods in news text are abbreviations or other non-terminal uses, not sentence boundaries [1]. Multiply this across dozens of languages, each with its own abbreviations, quotation conventions, and punctuation rules, and the scope of the problem becomes clear.

Rule-Based vs Machine Learning

SBD systems generally fall into two broad categories.

Rule-Based Systems

Rule-based systems encode linguistic knowledge directly. Rather than assuming every period ends a sentence, they examine context such as abbreviations, decimals, initials, URLs, quotation marks, list markers, and surrounding text before making a decision.

Strengths:

Fast: Minimal algorithmic overhead
Deterministic: Same input always yields the same boundaries
Explainable: Bugs can be traced to specific rules
Zero-shot: No training data required
Offline-friendly: Deploys easily on constrained hardware

Weaknesses:

Maintenance burden: Requires manual curation of abbreviation lists and rules
Brittleness: Fails on unforeseen formatting or domain shifts

Machine Learning Systems

ML models treat SBD as a classification problem, learning boundaries from annotated text.

Strengths:

Robust: Adapts to noisy, unpunctuated, or informal text
Contextual: Captures multi-token patterns without explicit rules
High accuracy: Frequently outperforms simple heuristics on benchmarks

Weaknesses:

Black box: Debugging requires retraining or fine-tuning
Resource-heavy: Higher inference latency and memory usage
Data-dependent: Requires annotated training data for each domain or language

This article focuses on rule-based systems, which remain the practical choice for many production pipelines due to their speed, determinism, and explainability.

Four Architectural Approaches

Rule-based splitters share a common goal but implement it very differently. Here are the four dominant architectures.

1. Mask → Split → Demask

Popularized by PySBD [2] and the Ruby Pragmatic Segmenter [3] it was ported from, this approach temporarily replaces ambiguous punctuation with placeholder tokens before splitting.

For example, the period in Dr. or example.com is masked. Once boundaries are identified, the original text is restored.

# Simplified example
def mask_split_restore(text):
    # Pass 1: mask abbreviation periods
    for abbr in ABBREVIATIONS:
        text = text.replace(f"{abbr}.", f"{abbr}∯")

    # Pass 2: split on remaining periods
    sentences = re.split(r'(?<=[.!?])\s+', text)

    # Pass 3: restore masked periods
    return [s.replace("∯", ".") for s in sentences]

Trade-offs:

Simple to implement and reason about
Memory-heavy: Each replacement allocates a new string; large documents can stress garbage collection [4]
Multiple passes: Requires scanning the text several times
Span reconstruction: Calculating character offsets relative to the original text requires post-processing matching [2]
Placeholder collisions: Must use characters that won't appear in the input

2. Rule-Based State Machines

Libraries such as spaCy's Sentencizer [5] and syntok [6] process text sequentially, applying contextual rules as they tokenize. Rather than masking punctuation, they maintain internal state and inspect neighboring tokens before declaring a boundary.

# Simplified example
def state_machine_segment(text):
    tokens = text.split()
    sentences, current = [], []

    for token in tokens:
        current.append(token)
        if (
            token.endswith(("!", "?"))
            or (
                token.endswith(".")
                and token.lower().rstrip(".") not in ABBREVIATIONS
            )
        ):
            sentences.append(" ".join(current))
            current = []

    if current:
        sentences.append(" ".join(current))
    return sentences

Trade-offs:

Integrates naturally with tokenization pipelines
Avoids modifying the input text
Architectural complexity: Every token passes through evaluation rules; as rule count grows, state transitions become difficult to maintain [7]
Tokenization dependency: Assumes tokenization is already correct

3. One-Pass Pointer-Based Parsing

A hardware-efficient approach that scans raw text once using a moving cursor. As characters are encountered, contextual lookaheads and lookbehinds determine whether a character constitutes a sentence boundary.

Because the input is never rewritten or rescanned, allocations are minimal. No placeholder management, no restore pass.

# Simplified example
def pointer_parse(text):
    boundaries = [0]
    i = 0

    while i < len(text):
        if text[i] in "!?":
            boundaries.append(i + 1)
        elif text[i] == ".":
            word_start = text.rfind(" ", 0, i) + 1
            word = text[word_start:i].lower()
            if word not in ABBREVIATIONS:
                boundaries.append(i + 1)
        i += 1

    return boundaries

Trade-offs:

Memory efficient: Single pass, no string duplication [8]
Hardware-friendly: Minimal allocations, good cache behavior
Code complexity: Adding rules requires threading lookahead logic into a nested loop
Error-prone pointer arithmetic: Handling nested quotes or ellipses requires care

4. Two-Pass Candidate Detection

This architecture (used by yasbd-lib [9]) flips the traditional logic. Instead of shielding text ahead of time, it aggressively identifies all potential boundaries first, then filters out false positives.

Pass 1: Find every position that could plausibly end a sentence: periods, question marks, exclamation points followed by whitespace, uppercase, or newline. Deliberately over-inclusive.
Pass 2: Evaluate each candidate against protection rules (abbreviation lists, decimal patterns, URL detection), discarding invalid targets.

# Simplified example
def two_pass_detect(text):
    # Pass 1: find ALL potential boundary candidates
    candidates = [m.start() for m in re.finditer(r'(?<=[.!?])(?=\s+[A-Z])', text)]

    # Pass 2: filter out invalid points
    real_boundaries = []
    for pos in candidates:
        word_start = text.rfind(" ", 0, pos) + 1
        if text[word_start:pos].lower() not in ABBREVIATIONS:
            real_boundaries.append(pos)

    return real_boundaries

Trade-offs:

Harder to miss a boundary: Over-inclusive by design [9]
Separates scanning from validation: Cleaner code organization
Synchronization risk: Pass 1 and Pass 2 must remain aligned
Requires careful candidate detection: Patterns must match what the filter can evaluate

Design Considerations

Choosing an architecture involves trade-offs across several dimensions.

Accuracy

All rule-based systems depend on the quality and completeness of their rule sets. The mask-then-split approach can achieve high accuracy on curated test sets, but its monolithic rule pipeline makes targeted fixes difficult. Two-pass systems isolate validation, making it easier to add or modify rules without affecting candidate detection [9].

Performance

Mask-then-split performs multiple passes and string allocations, which can become a bottleneck on large documents [4]. Pointer-based and two-pass approaches minimize allocations and are generally faster on modern hardware [8, 9].

Maintainability

State machines and pointer-based parsers embed logic in control flow, making rule additions invasive. Mask-then-split centralizes rules but creates interdependencies between transformations. Two-pass systems separate detection from filtering, simplifying rule management [9].

Span Tracking

If your pipeline needs character offsets into the original text (for NER training, RAG indexing, or translation alignment), consider how each architecture handles span reconstruction. Mask-then-split requires post-processing matching to map boundaries back to the original string [2]. Pointer-based and two-pass systems track positions directly, avoiding this step.

Multilingual Support

Rule-based SBD across languages requires per-language abbreviation lists, quotation rules, and punctuation conventions. Some architectures make this easier than others. PySBD supports 22 languages via modular language profiles [2]. yasbd-lib supports 39 languages using a declarative rule system that isolates language-specific data [9]. Two-pass systems with external language packs can add new languages without modifying core code [9].

Recommendations

There is no single "best" architecture—each has trade-offs suited to different constraints.

Constraint	Recommended Architecture
Simple implementation, modest text sizes	Mask → Split → Demask
Integration with existing tokenizers	State machine
Minimal memory usage, large documents	One-pass pointer-based
Ease of rule maintenance and extension	Two-pass candidate detection
Accurate character span tracking	Pointer-based or two-pass
Multiple languages	Modular language profiles (any architecture)

For new projects that require:

High accuracy on diverse text
Easy rule maintenance without breaking the pipeline
Native character span tracking
Multilingual support

...a two-pass or pointer-based architecture with modular language profiles is often the most sustainable choice.

For existing pipelines that already use a mask-then-split library, the question is whether the maintenance burden and performance characteristics are acceptable for your use case.

References

[1] E. Stamatatos, N. Fakotakis, and G. Kokkinakis. "Automatic extraction of rules for sentence boundary disambiguation." Proceedings of the Workshop on Machine Learning in Human Language Technology, University of Patras, pp. 88-92.

[2] N. Sadvilkar and M. Neumann. "PySBD: Pragmatic Sentence Boundary Disambiguation." arXiv:2010.09657, 2020. https://arxiv.org/abs/2010.09657

[3] Pragmatic Segmenter (Ruby). https://github.com/diasks2/pragmatic_segmenter

[4] A. M. Kuchling. "Python Performance Tips." Python Documentation. https://docs.python.org/3/howto/performance.html

[5] spaCy Sentencizer. https://spacy.io/api/sentencizer

[6] F. Leitner. "syntok: Text tokenization and sentence segmentation." https://github.com/fnl/syntok

[7] M. Fowler. Refactoring: Improving the Design of Existing Code. Addison-Wesley, 1999.

[8] J. Dean and S. Ghemawat. "MapReduce: Simplified Data Processing on Large Clusters." OSDI 2004.

[9] yasbd-lib Documentation. https://github.com/speedyk-005/yasbd-lib/

Yasbd-lib vs. PySBD: Higher Accuracy, More Languages, Faster Speed

Speedyk-005 — Sun, 19 Jul 2026 19:22:59 +0000

Sentence boundary detection sounds boring. Split on . ? !, done, right? Anyone who has tried knows otherwise. Abbreviations, decimals, URLs, nested quotes, ellipsis, legal citations, biomedical jargon—each one turns "split text" into a language-specific puzzle.

Two Python libraries tackle this problem with different philosophies. pysbd has been the go-to since 2020 with 23 languages, ported from Ruby's pragmatic segmenter [1]. yasbd-lib is newer, covers 39 languages, and takes a different architectural approach.

This is the difference between protecting boundaries and finding them.

Architecture: Mutation vs. Pointers

To understand how these engines behave on large datasets, we have to look at how they treat input strings.

PySBD: The Transformation Pipeline

pip install pysbd -U

PySBD operates as a multi-stage transformation pipeline [1]. It treats text as a mutable object that must be modified before it can be split [1]. To prevent punctuation within abbreviations, numbers, or URLs from triggering false splits, PySBD applies regular expressions to replace characters with placeholder tokens [1].

flowchart TD
    A[Raw Input Text] --> B["Replace . with {} / Mask URLs"]
    B --> C[Run Rule Engine]
    C --> D[Split on Splitting Marks]
    D --> E[Reverse Replacement / Restore Text]
    E --> F[Extract Text Segments]

The structural consequence: the original string layout is transformed during processing. Because the text is modified mid-flight, calculating exact character offsets (spans) relative to the original uncleaned text requires a post-processing reconstruction step [1]. If you enable text cleaning (clean=True), PySBD raises an error when requesting character spans because it cannot guarantee coordinate matching after modification [1].

yasbd-lib: The Query Planning Approach

pip install yasbd-lib -U

yasbd-lib treats text as immutable [2]. It does not modify the raw string [2]. Instead, its architecture resembles a database query planner—generating candidate coordinate arrays and using language-specific filters to narrow down boundary slices [2].

flowchart LR
    A[Raw Input String] --> B[Pass 1: Aggressive Candidate Identification]
    B --> C[Pass 2: Modular Filter Elimination]
    C --> D[Project Slices / Return Index Pointers]

By evolving integer pointers rather than altering text strings, yasbd-lib maintains context of the source layout throughout processing [2]. Token spans are tracked as a first-class structural signal during parsing rather than reconstructed afterward [2].

Deep-Dive Feature Breakdown

1. Spans and Character Offsets

Because PySBD does not natively track indices during its transformation phase, calculating character offsets requires a post-processing step that searches the original document to locate each sentence [1].

The PySBD Reconstruction Step

PySBD reconstructs spans by scanning the original text for each sentence (source):

def sentences_with_char_spans(self, sentences):
    sent_spans = []
    prior_end_char_idx = 0
    for sent in sentences:
        for match in re.finditer('{0}\s*'.format(re.escape(sent)), self.original_text):
            match_str = match.group()
            match_start_idx, match_end_idx = match.span()
            if match_end_idx > prior_end_char_idx:
                sent_spans.append(
                    TextSpan(match_str, match_start_idx, match_end_idx))
                prior_end_char_idx = match_end_idx
                break
    return sent_spans

The Trade-off: This reconstruction performs repeated searches over the original document. In pathological cases—such as documents with many repeated sentences—this can approach quadratic behavior. For typical use cases, the overhead is manageable, but it does add runtime cost on longer texts.

The yasbd-lib Approach

In yasbd-lib, spans are produced natively during boundary detection [2]. It emits boundary allocations dynamically, avoiding lookback overhead [2]. The library also provides an adapter layer for migrating from PySBD [2]:

from yasbd.utils.pysbd_adapter import Segmenter

seg = Segmenter(language="ja")
res = seg.segment('田中さんは「準備は完了しました」そう言って部屋を出た。U.S.A.の経済政策 is complex.')
print(res)
# ['田中さんは「準備は完了しました」そう言って部屋を出た。', 'U.S.A.の経済政策 is complex.']

2. Memory and Streaming

PySBD processes text as complete string buffers [1]. yasbd-lib provides abstractions for memory-constrained environments through lazy evaluation via ParagraphStream and StreamCleaner [2]:

from yasbd.utils.cleaner import StreamCleaner
from yasbd import BoundaryDetector

cleaner = StreamCleaner("Hello  world.   This is  messy.")
detector = BoundaryDetector(lang="en")

sentences = list(detector.segment(cleaner))
print(sentences)
# ['Hello world.', 'This is messy.']

Since v0.14.0 the default cleaning pipeline also normalizes \r\n and \r line endings to \n and rejoins hyphenated words split across line breaks (work-\ning becomes working) while keeping legitimate hyphenated compounds intact [2].

3. Resource Management

Under the hood of the BoundaryDetector pipeline, yasbd-lib manages execution rules using a 5-entry LRU cache (_MAX_CACHED_RULES = 5) [2]. When using automatic language identification (lang="auto"), if confidence drops below the threshold (_MIN_CONFIDENCE = 0.8), the module logs an informational message rather than masking the failure [2]:

# From boundary_detector.py
if lang == "auto":
    lang, confidence = classify_language(snippet)
    if confidence < _MIN_CONFIDENCE:
        log_info(
            self.verbose,
            "Low confidence ({:.2f}) for detected lang {!r} in auto mode",
            confidence,
            lang,
        )

Additionally, yasbd-lib supports preserving token boundaries inside parentheses or brackets via preserve_quote_and_paren=True [2].

Maintenance Status: A Critical Consideration

The architectural differences matter, but there's another factor: PySBD is effectively unmaintained. As of December 2025, an open issue (#135) [9] notes that the repository has seen no recent updates, with multiple open PRs from contributors and a maintainer who has seemingly abandoned the project. The issue author explicitly requested archiving the project to signal to downstream users that they should no longer incorporate it [9].

The maintenance situation has real consequences. Consider these unresolved issues:

[Issue #79] [10] - Infinite Loop (October 2020):

segmenter = pysbd.Segmenter(language="en", clean=False)
text = "..[111 111 111 111 111 111 111 111 111 111]"
segmenter.segment(text)  # Hangs indefinitely

The problem is catastrophic backtracking in NUMBERED_REFERENCE_REGEX. The maintainer acknowledged it in February 2021, saying "Need to dug into details" [10]. Over four years later, it remains unresolved.

[Issue #92] [11] - Catastrophic Backtracking in HTMLTagRule (February 2021):

HTMLTagRule = Rule(r"<\/?\w+((\s+\w+(\s*=\s*(?:\".*?\"|'.*?'|[\^'\">\s]+))?)+\s*|\s*)\/?>", '')

When processing unfinished HTML attributes, this regex can cause the segmenter to hang indefinitely [11]. A simplified fix was proposed in the same issue, but it remains unreviewed and unmerged.

Both issues stem from the same root cause: regex patterns with nested quantifiers in the transformation pipeline [10, 11]. The project has no active maintainer to review or merge fixes [9].

yasbd-lib was built in response to this situation, offering a drop-in adapter for PySBD to fix edge cases without heavy refactoring [2, 9].

Benchmark Comparisons

The architectural differences influence accuracy and performance.

Feature	PySBD	yasbd-lib
Python Support	3.7–3.11 [1]	3.10–3.14 [2]
Maintenance	Unmaintained (as of 2025) [9]	Actively maintained [2]
Known Issues	Infinite loop on numbered references [10]; catastrophic backtracking in HTML cleaner [11]	No known catastrophic backtracking issues
Approach	Monolithic transformation pipeline [1]	Modular immutable pipeline [2]
State Handling	String mutation with placeholder tokens [1]	Pointer-based operations [2]
Language Profiles	23 Languages [1]	39 Languages [2]
English Golden Score	77 / 92 (83.7%) [2]	91 / 92 (98.9%) [2]
Framework Adapters	Native API [1]	spaCy v3+ integration [2]

Benchmark Note: The English Golden Score is measured on the project's expanded golden corpus of 92 evaluation cases [2]. The original PySBD corpus contained 48 cases; the expanded set removes ambiguous examples and adds coverage for abbreviation chains, contiguous terminators, and other edge cases. Full methodology and test cases are available in the benchmarks directory.

A multi-library performance comparison across increasing text sizes. yasbd-lib consistently outperforms alternatives at every scale.

Edge Case Behavior

Consider how both engines handle challenging inputs:

Input with numbered references (Issue #79): "..[111 111 111 111 111 111 111 111 111 111]" [10]

PySBD: Can enter an infinite loop due to catastrophic backtracking in NUMBERED_REFERENCE_REGEX. This was reported in October 2020 and remains unresolved [10].
yasbd-lib: The two-pass boundary detection approach avoids complex regex substitutions, preventing this class of issue [2].

Input with unfinished HTML (Issue #92): "<iframe width="100%" ... src="url Lorem ipsum..." [11]

PySBD: The HTML cleaning regex can cause catastrophic backtracking, hanging the segmenter indefinitely [11]. Reported in February 2021, still unresolved.
yasbd-lib: Uses a StreamCleaner with configurable cleaning steps, including optional HTML unwrapping that avoids nested quantifiers [2].

Extensibility: Configuration Approaches

What happens when you need to handle custom abbreviations like "Com." or "Adm."?

PySBD: Internal Rule Modification

Because PySBD's rules operate on a shared transformation timeline, they are interdependent [1]. Adding exceptions requires modifying the internal mutation flow [1].

As documented in [Issue #108] [3]:

"Unfortunately, there is no specific documentation about modifying rules as there are so many and each rule is associated with some form of transformation... All those operations need to be performed in that sequence as they are interrelated... Best way is to use python debugger and see how your input text goes through different transformations."

Adding rules without understanding the full pipeline can break downstream regex patterns [1]. With the project unmaintained, there is no clear path for getting such fixes merged upstream [9].

yasbd-lib: Declarative Language Profiles

yasbd-lib decouples matching mechanics from language-specific data [2]. It exposes structured hooks for customization [2]:

The base Rules class defines sets for:

TITLE_ABBRVS: Honorifics that should not split sentences
REFERENCE_ABBRVS: Citation abbreviations (fig, pág)
INLINE_ONLY_ABBRVS: Abbreviations that don't end sentences (blvd)
DATE_ABBRVS: Month and weekday abbreviations
DOTTED_GEOPOL_ABBRVS: Geo abbreviations like U.S., E.U.
TERMINATORS: Extra sentence-ending punctuation
COMMON_SENT_STARTERS: Boundary hints for languages without spaces
POST_QUOTATIVE_PARTICLES and REPORTING_WORDS: For dialogue attribution

To add a new language, you create a file like fr.py, subclass Rules as FrRules, and override only the sets your language needs. The language template [6] provides the structure.

Post-Processing Hook

Since v0.14.0, BoundaryDetector accepts a hook callback that runs per paragraph after the language rules apply [2]. It receives a dict with text, lang, boundaries and paragraph_index and can add or remove sentence boundaries in place. This replaces monkey-patching rule internals for custom boundary logic:

from yasbd import BoundaryDetector

def add_extra_boundaries(ctx):
    # ctx: {text, lang, boundaries, paragraph_index}
    pass  # mutate ctx["boundaries"] in place

detector = BoundaryDetector(lang="zh", hook=add_extra_boundaries)

PySBD offers no equivalent: custom boundary logic there means modifying the internal mutation flow [3].

External Language Packs

yasbd-lib supports loading custom language modules at runtime via register_lang_packs() [2]:

from yasbd.rules import register_lang_packs
from yasbd import BoundaryDetector

register_lang_packs(["clinical_yasbd_pack"])
detector = BoundaryDetector(lang="clinical")

Changes to a language-specific profile do not affect the core engine's boundary detection [2].

Language Profile Policy

As documented in [Issue #198] [5], yasbd-lib has frozen its built-in language set at 39 profiles for the v1.x series to maintain API stability. Additional languages must be loaded externally via register_lang_packs() using community-maintained packages like yasbd-union or yasbd-auxlang.

Which Library Should You Choose?

graph TD
    A[Which SBD to choose?] --> B{Using legacy spaCy v2?}
    B -- Yes --> C[Consider PySBD with caution]
    B -- No --> D{Need active maintenance?}
    D -- Yes --> E[Use yasbd-lib]
    D -- No --> F[Use yasbd-lib for accuracy gains]

Consider PySBD only if:

Absolute legacy lock-in: You are maintaining an existing pipeline tied to spaCy v2 or older deployments that cannot be migrated. Be aware that the project is unmaintained and has known unresolved issues, including infinite loops with numbered references and catastrophic backtracking with certain HTML inputs [9, 10, 11].

Use yasbd-lib if:

Active maintenance: The project is actively maintained with a clear contribution path [2].
Performance at scale: Benchmark results on the Sherlock Holmes text (594k characters) show yasbd completing in approximately 1.6 seconds compared to 13.3 seconds for PySBD on the same hardware [2]. (These results are from the project's benchmark suite; your mileage may vary based on hardware and Python version.)
Character span accuracy: Native span tracking may be preferable for downstream tasks like NER training or RAG indexing [2].
Non-standard inputs: The modular design handles raw markdown, chat logs, and multilingual text [2].
Custom language rules: The declarative profile system simplifies adding new languages or domain-specific abbreviations [2].
Migration path: The included PySBD adapter allows incremental migration without rewriting your entire pipeline [2].
No catastrophic backtracking: The pointer-based architecture avoids the regex issues that plague PySBD's transformation pipeline [2, 10, 11].

References

Introducing chunklet-py v2.2.0+: One Library to Split Them All - Sentence, Code, Docs

Speedyk-005 — Mon, 23 Feb 2026 03:10:16 +0000

The Smart Text Chunking Library You Didn't Know You Needed

Ever tried splitting text for your RAG pipeline and ended up with chunks that cut sentences in half? Or worse — chunks that lose all context between them?

Yeah, I've been there too. That's exactly why I built chunklet-py — a Python library that actually understands text structure.

This post hits only the highlights and doesn't cover everything — visit the full documentation for everything else, including:

Custom sentence splitters for specialized languages
Custom document processors for unusual file formats
Custom tokenizers to match your LLM
The rich metadata you can get.
CLI flags for batch processing, parallel jobs, error handling, timeouts
Additional args like n_jobs, lang, show_progress, ...

⚠ Quick heads up!

This tutorial requires chunklet-py v2.2.0+ and uses APIs not available in earlier versions.

Upgrade to the latest version and see the documentation or What’s New for details.

The Problem with Dumb Splitting

Here's what usually happens:

# The naive approach
chunks = [text[i:i+500] for i in range(0, len(text), 500)]

This works... until it doesn't:

Sentences cut mid-way
No context between chunks
Broken code if you're chunking source files

Solution: chunklet-py

A smart text and code chunking library that respects natural boundaries.

Features

50+ languages supported — Auto-detects language and applies the right splitting rules. No more treating German the same as English.

Multiple constraint types — Mix and match:

max_sentences — group by sentences
max_tokens — respect LLM context limits
max_section_breaks — keep Markdown headers together (headings ##, horizontal rules ---, <details> tags)
max_lines — for code chunking
max_functions — keep functions together

Multiple file formats — PDF,PPTX, EML, DOCX, EPUB, HTML, Markdown, LaTeX, ODT, CSV, Excel, plain text — one library handles them all.

Rich metadata — Every chunk comes with source references, character spans, and structural info.

Composable constraints — Mix and match limits to get exactly the chunks you need.

Pluggable architecture — Swap in custom tokenizers, sentence splitters, or document processors.

What's New in v2.2.0

API Unification — Methods renamed to chunk_text, chunk_file, chunk_texts, chunk_files for consistency
Visualizer redesign — Fullscreen mode, 3-row layout, smoother hovers
More code languages — ColdFusion, VB.NET, PHP 8 attributes, Pascal support
Ruff — Switched to Ruff for faster linting

Check the What's New page for full details.

Installation

pip install chunklet-py

For document support:

pip install chunklet-py[structured-document]

For code:

pip install chunklet-py[code]

For visualization:

pip install chunklet-py[visualization]

Code Examples

Core Imports

from chunklet import DocumentChunker   # For PDFs, DOCX, and general text
from chunklet import CodeChunker       # For source code
from chunklet import SentenceSplitter  # For just sentences
from chunklet import visualizer        # Web-based visualizer

DocumentChunker API

Four methods cover most use cases:

Method	Input	Return Type
`chunk_text(text)`	str	List[Chunk]
`chunk_file(path)`	Path or str	List[Chunk]
`chunk_texts(list)`	List[str]	Generator[Chunk]
`chunk_files(list)`	List[Path]	Generator[Chunk]

DocumentChunker Example

chunker = DocumentChunker()

# Feel free to mix and match these
chunks = chunker.chunk_text(
    text,
    max_sentences=3,       # Stop after X sentences
    max_tokens=500,        # Don't blow up the LLM context
    max_section_breaks=2,  # Respect the Markdown headers
    overlap_percent=20,    # Give it some "memory" of the last chunk
    offset=0               # Skip the first N sentences
)

CodeChunker Example

chunker = CodeChunker()

chunks = chunker.chunk_text(
    code,
    max_lines=50,          # Height limit
    max_tokens=512,        # Width limit
    max_functions=1,       # One function per chunk
    strict=True,            # True: Crash on big blocks; False: Slice anyway
    include_comments=True,  # True by default
    docstring_mode="all",   # Options are: all, excluded, summary
)

⚠ Token Counter Requirement
When using the max_tokens constraint, a token_counter function is essential. This function, which you provide, should accept a string and return an integer representing its token count. Failing to provide a token_counter will result in a MissingTokenCounterError.
You can also provide the token_counter directly to any chunking method. If provided in both the constructor and the method, the one in the method will be used.

SentenceSplitter (Just Sentences)

from chunklet import SentenceSplitter

splitter = SentenceSplitter()
sentences = splitter.split_text(text, lang="en")   # You can also set it to "auto"

Handles tricky cases like "Dr." or "U.S.A." without breaking them up.

50+ languages are explicitly supported through dedicated libraries (pysbd covers 40+, Indic NLP Library covers 11, sentsplit covers 4, and Sentencex covers ~15, with some overlap), plus the Fallback Splitter handles any other language via Unicode rules (Supported Languages Documentation).

Output Object

Chunkers return Chunk objects (Box instances), so you use dot notation:

for chunk in chunks:
    print(chunk.content)   # The actual text/code
    print(chunk.metadata)  # Chunk metadata

Visualizer (Interactive Web UI)

Launch a web interface to experiment with chunking parameters:

chunklet visualize

Or programmatically:

from chunklet import visualizer

v = visualizer.Visualizer(host="127.0.0.1", port=8000)
v.serve()  # Opens in your browser

CLI Examples

Prefer the terminal? chunklet-py ships with a full CLI

Here are some quick examples:

# Basic text chunking
chunklet chunk "Your text here." --max-tokens 500

# Chunk a file
chunklet chunk --source document.pdf --max-tokens 500 --metadata

# Split text into sentences
chunklet split "Your text here." --lang en

# Split a file into sentences
chunklet split --source my_file.txt --destination sentences.txt

# Start the interactive visualizer
chunklet visualize

# Code chunking
chunklet chunk --code --source my_script.py --max-functions 1

# Batch processing a directory
chunklet chunk --doc --source ./my_docs --destination ./chunks --n-jobs 4

# With error handling
chunklet chunk --doc --source ./my_docs --on-errors skip

How It Compares

While there are other chunking libraries available, Chunklet-py stands out for its unique combination of versatility, performance, and ease of use. Here's a quick look at how it compares to some of the alternatives:

Library	Key Differentiator	Focus
chunklet-py	All-in-one, lightweight, multilingual, language-agnostic with specialized algorithms.	Text, Code, Docs
LangChain	Full LLM framework with basic splitters (e.g., RecursiveCharacterTextSplitter, Markdown, HTML, code splitters). Good for prototyping but basic for complex docs or multilingual needs.	Full Stack
Chonkie	All-in-one pipeline (chunking + embeddings + vector DB). Uses `tree-sitter` for code. Multilingual.	Pipelines
Semchunk	Text-only, fast semantic splitting. Built-in tiktoken/HuggingFace support. 85% faster than alternatives.	Text
CintraAI Code Chunker	Code-specific, uses `tree-sitter`. Initially supports Python, JS, CSS only.	Code

Chunklet-py is a specialized, drop-in replacement for the chunking step in any RAG pipeline. It handles text, documents, and code without heavy dependencies, while keeping your project lightweight.

Stop Breaking Context: Smarter Text Chunking for Python NLP Projects

Speedyk-005 — Wed, 13 Aug 2025 21:59:51 +0000

Chunklet: Smarter Text Chunking for Python Developers

⚠ This post is outdated

This guide uses chunklet v1.x, which is no longer maintained. see the Migration Guide: https://speedyk-005.github.io/chunklet-py/latest/migration/

👉 Use chunklet-py v2.x instead:

https://dev.to/speed_k_7e1b449706e59e433/chunklet-py-one-library-to-split-them-all-sentence-code-docs-2eeg

🚀 Latest version (v2.2.0+):

https://dev.to/speed_k_7e1b449706e59e433/-introducing-chunklet-py-dj8

Why Context Matters in Text Splitting

When preprocessing documents for NLP tasks, standard splitting methods often:

Break sentences mid-thought ("The patient showed improvement. However," → "However,")
Ignore linguistic boundaries in non-English texts
Lose critical context between chunks

Chunklet solves this with structural awareness.

1. Installation & Basic Usage

pip install chunklet

Minimal Example:

from chunklet import Chunklet

text = "First sentence. Second sentence. Third sentence."
chunker = Chunklet()
chunks = chunker.chunk(text, mode="sentence", max_sentences=2)

# Output:
# ["First sentence. Second sentence.", "Third sentence."]

This preserves complete sentences while respecting chunk size limits.

2. Key Features Explained

Hybrid Chunking Mode

Combines structural and size-based splitting:

chunks = chunker.chunk(
    text,
    mode="hybrid",
    max_sentences=3,  # Structural limit
    max_tokens=200,   # Size limit
    overlap_percent=15  # Context preservation
)

Why this matters:

Prevents chunks from becoming too long or too short
Overlap maintains relationships between sections
Works equally well on code, markdown, or prose

Multilingual Support

# Auto-detection (36+ languages)
chunks = chunker.chunk(multilingual_text)

# Manual override
chunks = chunker.chunk(japanese_text, language="ja")

How it works:

Uses py3langid for fast language detection
Applies language-specific sentence boundaries
Falls back to regex for unsupported languages

3. Real-World Use Cases

Preparing Legal Documents

legal_text = Path("contract.txt").read_text()
chunks = chunker.chunk(
    legal_text,
    mode="hybrid",
    max_tokens=512,
    overlap_percent=20  # Critical for clause relationships
)

Why it works:

Preserves entire contract clauses
Maintains references between sections (e.g., "as defined in Section 2.1")
Handles complex punctuation in legal prose

Processing Academic Papers

chunker = Chunklet(
    sentence_splitter=custom_academic_splitter,  # Handles citations
    token_counter=scibert_tokenizer  # Domain-specific counting
)

Customization options:

Plug in any sentence splitter
Use HuggingFace tokenizers
Adjust chunking thresholds per document type

4. Performance Considerations

# For large datasets:
results = chunker.batch_chunk(
    documents,
    n_jobs=4,          # Parallel processing
    chunk_size=1000     # Documents per batch
)

Optimization tips:

Enable use_cache=True for repeated texts
Pre-filter very short/long documents
Monitor memory with memory_profiler

Ready to try?

GitHub Repository | PyPI Package