Replacing spaCy's Sentencizer with yasbd: From 55.4% to 98.9% Accuracy

#nlp #rulebased #python #sentence

Sentence segmentation—splitting text into individual sentences—is a foundational step in most NLP pipelines. It seems simple, but as anyone who has worked with real-world text knows, the details matter. Abbreviations like Dr., URLs, ellipsis, and complex punctuation can trip up even well-intentioned rule-based systems.

spaCy's built-in Sentencizer is a fast, lightweight, rule-based component designed for this task. However, by design, it makes a trade-off: speed and simplicity over accuracy on edge cases. For many production applications, this trade-off leads to catastrophic downstream errors.

Enter yasbd (Yet Another Sentence Boundary Detector). It's a pure-Python, rule-based SBD that can be integrated as a drop-in spaCy component, and it achieves a remarkable 98.9% accuracy on a challenging English golden benchmark, compared to spaCy's Sentencizer which scores only 55.4% [1].

This post will walk you through why spaCy's default segmenter falls short, and how you can easily upgrade your pipeline with yasbd to achieve near-perfect sentence boundaries.

Why spaCy's Sentencizer is Intentionally Lightweight

The Sentencizer is a non-trainable pipeline component. Reading its source code reveals its core logic is only a few dozen lines [2]:

Mark the first token of the document as a sentence start.
Walk through every token in the document.
If a token is in punct_chars (., ?, !, 。, etc.), remember that a sentence may end here.
Skip over any following punctuation.
As soon as the next token is not punctuation, mark it as the beginning of a new sentence.

In simplified pseudocode, it looks like this:

start = 0
seen_period = False
doc_guesses[0] = True

for token in doc:
    if token.text in punct_chars:
        seen_period = True
    elif seen_period:
        doc_guesses[token.i] = True
        seen_period = False

That's it. It performs no abbreviation list lookup, no context analysis, and no handling of quotes, parentheses, or URLs. As the spaCy documentation itself notes, it's a component for when you "don't require the dependency parse" and want a "simpler, rule-based strategy" [3]. It was never designed to be a robust sentence splitter for messy, real-world text.

Wait, Doesn't spaCy Handle Abbreviations?

You might be wondering: "But spaCy handles Dr. just fine, doesn't it?" This is a sharp observation, and it gets to an important distinction in how spaCy processes text.

The Sentencizer itself has no abbreviation awareness, but spaCy's Tokenizer runs first and applies hard-coded rules called Tokenizer Exceptions for common patterns like Dr., Mr., Ms., and U.K. [5, 6].

Here's what happens:

· Dr.: The Tokenizer keeps it as a single token ['Dr.']. The Sentencizer sees this as one unit and often bypasses the boundary check.
· A.: The Tokenizer splits this into 'A', '.'. The Sentencizer now sees a standalone period token and forces a sentence split.
· M.D.: No built-in exception exists, so the Tokenizer splits it into ['M', '.', 'D', '.']. The Sentencizer sees multiple standalone periods and fragments the text.

This is why the Sentencizer works on simple cases like "Dr. Smith" but collapses on compound abbreviations, citations, or text with multiple initials. The tokenizer provides some protection, but it's incomplete and inconsistent [4, 6].

A GitHub issue from 2021 highlights this limitation: a user pointed out that for Polish, the Sentencizer doesn't handle common abbreviations like dr., prof., or inż., making it effectively broken for that language [6]. The response from the spaCy team acknowledged that adding default exceptions for all languages is challenging, as the tokenizer defaults follow the guidelines of specific training corpora [7]. This means that for the vast majority of the 75+ languages it claims to support, the Sentencizer has minimal abbreviation awareness.

The Benchmark Gap: 55.4% vs. 98.9%

The performance difference becomes stark when you test these tools on a diverse set of challenging English texts. The yasbd benchmark suite includes a golden dataset of 92 English edge cases, expanded from the original 48 cases used by pysbd. These cases are specifically designed to break naive splitters and include abbreviation chains, URLs, quotes, ellipsis, and academic citations [1].

The results from the yasbd benchmarks are clear [1]:

Library	Score on 92 Edge Cases
yasbd	91/92 (98.9%)
pysbd	77/92 (83.7%)
sentencex	76/92 (82.6%)
blingfire	75/92 (81.5%)
sentsplit	61/92 (66.3%)
sentence-splitter	60/92 (65.2%)
nupunkt	59/92 (64.1%)
spacy-sentencizer	51/92 (55.4%)

This puts spacy-sentencizer at the very bottom of the list. It fails on nearly half of the test cases, primarily because it has no concept of abbreviations and will blindly split on any period, regardless of context.

How to Install yasbd-lib

The installation is straightforward via pip. Note that yasbd-lib is a pure-Python library with no native dependencies, making it easy to install in any environment.

pip install yasbd-lib -U

That's it.

Integrating yasbd as a spaCy Component

This is where the power of yasbd really shines for spaCy users. It provides a simple one-function registration API, allowing you to replace the default sentencizer with just a few lines of code [8].

spaCy is not a dependency of yasbd. You will need to install it separately.

pip install spacy -U

Once you have both libraries, the integration is seamless. The key is to add the yasbd component first in your pipeline. This ensures its sentence boundaries are set early and can be used by subsequent components like the parser or NER, rather than being overwritten.

import spacy
from yasbd import register_spacy_component

# 1. Register the component. This tells spaCy about the "yasbd" factory.
register_spacy_component()  # Requires spaCy v3+

# 2. Create a blank English pipeline.
nlp = spacy.blank("en")

# 3. Add the yasbd component FIRST.
#    `first=True` ensures it runs before the parser, so its boundaries are preserved.
nlp.add_pipe("yasbd", first=True, config={"lang": "en"})

# 4. Process your text.
doc = nlp("Dr. Smith arrived. He was late.")

# 5. Access the beautifully segmented sentences.
for sent in doc.sents:
    print(sent.text)
# Output:
# Dr. Smith arrived.
# He was late.

Configuration

The yasbd component is highly configurable. You can set the language, enable verbose logging, or even use automatic language detection [8].

# Language can be inherited from the pipeline's language
nlp.add_pipe("yasbd", first=True)  # `lang` defaults to `nlp.lang`

# Or, use auto-detection (slower, but useful for mixed-language text)
nlp.add_pipe("yasbd", first=True, config={"lang": "auto"})

# Or, configure it at runtime
pipe = nlp.get_pipe("yasbd")
pipe.detector.lang = "fr"  # Switch to French rules
pipe.detector.verbose = True
pipe.preserve_quote_and_paren = False  # Disable quote/parenthesis protection

Before and After Examples

Let's see the difference in action on a piece of text designed to break the Sentencizer.

import spacy
from yasbd import register_spacy_component

register_spacy_component()

# --- Pipeline with spaCy's default sentencizer ---
nlp_default = spacy.blank("en")
nlp_default.add_pipe("sentencizer")

# --- Pipeline with yasbd ---
nlp_yasbd = spacy.blank("en")
nlp_yasbd.add_pipe("yasbd", first=True, config={"lang": "en"})

text = """Dr. Patel A. (M.D., Ph.D.), can corroborate my claim.
You can reach me at j.doe@example.com or visit https://www.example.com/page?ref=1.
As Smith et al. (2021) noted: "The implications are far-reaching." However, critics disagree."""

print("--- spaCy Sentencizer ---")
for sent in nlp_default(text).sents:
    print(f"'{sent.text}'")
# Output:
# 'Dr. Patel A. (M.D., Ph.D.), can corroborate my claim.'
# '
# You can reach me at j.doe@example.com or visit https://www.example.com/page?ref=1.'
# '
# As Smith et al. ('
# '2021) noted: "The implications are far-reaching."'
# 'However, critics disagree.'
#
# Notice: Newlines become sentence starts, and "2021" becomes its own fragment.

print("\n--- yasbd ---")
for sent in nlp_yasbd(text).sents:
    print(f"'{sent.text}'")
# Output:
# 'Dr. Patel A. (M.D., Ph.D.), can corroborate my claim.
# '
# 'You can reach me at j.doe@example.com or visit https://www.example.com/page?ref=1.
# '
# 'As Smith et al. (2021) noted: "The implications are far-reaching."'
# 'However, critics disagree.'

The Sentencizer output is a mess:

· It splits on newline characters, creating empty sentence fragments.
· It shatters the citation "(2021)" into its own sentence.
· It completely fails to recognize compound abbreviations like M.D., Ph.D..

yasbd, with its comprehensive two-pass system—first identifying candidate boundaries, then surgically removing false positives using abbreviation lists and context rules—handles the entire text flawlessly [8].

Conclusion

Sentence segmentation is a critical first step in any NLP pipeline. While spaCy's default Sentencizer is a convenient, lightweight option, its simplistic, period-based logic makes it highly inaccurate on real-world text. While the Tokenizer provides some protection for common abbreviations like Dr., this protection is incomplete and fails on compound abbreviations, initials, and non-English text [5, 6, 7]. The benchmarks show it failing on nearly half of the tested edge cases [1].

yasbd offers a compelling alternative. It is a high-accuracy, rule-based SBD that can be integrated as a spaCy component with just a few lines of code, providing a massive leap in accuracy from 55.4% to 98.9% [8].

For any spaCy user who cares about the quality of their sentence boundaries—and the downstream tasks that depend on them—replacing the sentencizer with yasbd is an easy and highly effective upgrade.