DEV Community

Cover image for Four ways to split sentences (without machine learning)
Speedyk-005
Speedyk-005

Posted on

Four ways to split sentences (without machine learning)

#nlp #architecture #regex #rulebased

Before a machine can summarize text, translate documents, extract entities, or answer questions, it first needs to know where each sentence begins and ends. Get the boundaries wrong, and downstream tasks inherit corrupted input.

This article explores the design space for rule-based sentence boundary disambiguation (SBD) systems, examining the architectural choices that determine correctness, performance, and maintainability.

The Problem: Punctuation Is Overloaded

The obvious solution seems almost embarrassingly simple:

re.split(r'(?<=[.!?])(?=\s+[A-Z])', text)
Enter fullscreen mode Exit fullscreen mode

It works... until it doesn't. A period alone does six jobs and only one is "sentence end." Here are just a few situations where naive regex fails:

Input Why it fails
Dr. Smith arrived. Dr. is an abbreviation, not a sentence end
Apple Inc. announced... Inc. isn't the end
The value is 3.14159. Decimal point
D. H. Lawrence wrote... Initials
Wait... what? Ellipsis
Version 2.5.1 fixes bugs. Software version
Visit example.com. Domain name
1. Install Python. Numbered list

Studies estimate that approximately 47% of periods in news text are abbreviations or other non-terminal uses, not sentence boundaries [1]. Multiply this across dozens of languages, each with its own abbreviations, quotation conventions, and punctuation rules, and the scope of the problem becomes clear.

Rule-Based vs Machine Learning

SBD systems generally fall into two broad categories.

Rule-Based Systems

Rule-based systems encode linguistic knowledge directly. Rather than assuming every period ends a sentence, they examine context such as abbreviations, decimals, initials, URLs, quotation marks, list markers, and surrounding text before making a decision.

Strengths:

  • Fast: Minimal algorithmic overhead
  • Deterministic: Same input always yields the same boundaries
  • Explainable: Bugs can be traced to specific rules
  • Zero-shot: No training data required
  • Offline-friendly: Deploys easily on constrained hardware

Weaknesses:

  • Maintenance burden: Requires manual curation of abbreviation lists and rules
  • Brittleness: Fails on unforeseen formatting or domain shifts

Machine Learning Systems

ML models treat SBD as a classification problem, learning boundaries from annotated text.

Strengths:

  • Robust: Adapts to noisy, unpunctuated, or informal text
  • Contextual: Captures multi-token patterns without explicit rules
  • High accuracy: Frequently outperforms simple heuristics on benchmarks

Weaknesses:

  • Black box: Debugging requires retraining or fine-tuning
  • Resource-heavy: Higher inference latency and memory usage
  • Data-dependent: Requires annotated training data for each domain or language

This article focuses on rule-based systems, which remain the practical choice for many production pipelines due to their speed, determinism, and explainability.

Four Architectural Approaches

Rule-based splitters share a common goal but implement it very differently. Here are the four dominant architectures.

1. Mask → Split → Demask

Popularized by PySBD [2] and the Ruby Pragmatic Segmenter [3] it was ported from, this approach temporarily replaces ambiguous punctuation with placeholder tokens before splitting.

For example, the period in Dr. or example.com is masked. Once boundaries are identified, the original text is restored.

# Simplified example
def mask_split_restore(text):
    # Pass 1: mask abbreviation periods
    for abbr in ABBREVIATIONS:
        text = text.replace(f"{abbr}.", f"{abbr}∯")

    # Pass 2: split on remaining periods
    sentences = re.split(r'(?<=[.!?])\s+', text)

    # Pass 3: restore masked periods
    return [s.replace("∯", ".") for s in sentences]
Enter fullscreen mode Exit fullscreen mode

Trade-offs:

  • Simple to implement and reason about
  • Memory-heavy: Each replacement allocates a new string; large documents can stress garbage collection [4]
  • Multiple passes: Requires scanning the text several times
  • Span reconstruction: Calculating character offsets relative to the original text requires post-processing matching [2]
  • Placeholder collisions: Must use characters that won't appear in the input

2. Rule-Based State Machines

Libraries such as spaCy's Sentencizer [5] and syntok [6] process text sequentially, applying contextual rules as they tokenize. Rather than masking punctuation, they maintain internal state and inspect neighboring tokens before declaring a boundary.

# Simplified example
def state_machine_segment(text):
    tokens = text.split()
    sentences, current = [], []

    for token in tokens:
        current.append(token)
        if (
            token.endswith(("!", "?"))
            or (
                token.endswith(".")
                and token.lower().rstrip(".") not in ABBREVIATIONS
            )
        ):
            sentences.append(" ".join(current))
            current = []

    if current:
        sentences.append(" ".join(current))
    return sentences
Enter fullscreen mode Exit fullscreen mode

Trade-offs:

  • Integrates naturally with tokenization pipelines
  • Avoids modifying the input text
  • Architectural complexity: Every token passes through evaluation rules; as rule count grows, state transitions become difficult to maintain [7]
  • Tokenization dependency: Assumes tokenization is already correct

3. One-Pass Pointer-Based Parsing

A hardware-efficient approach that scans raw text once using a moving cursor. As characters are encountered, contextual lookaheads and lookbehinds determine whether a character constitutes a sentence boundary.

Because the input is never rewritten or rescanned, allocations are minimal. No placeholder management, no restore pass.

# Simplified example
def pointer_parse(text):
    boundaries = [0]
    i = 0

    while i < len(text):
        if text[i] in "!?":
            boundaries.append(i + 1)
        elif text[i] == ".":
            word_start = text.rfind(" ", 0, i) + 1
            word = text[word_start:i].lower()
            if word not in ABBREVIATIONS:
                boundaries.append(i + 1)
        i += 1

    return boundaries
Enter fullscreen mode Exit fullscreen mode

Trade-offs:

  • Memory efficient: Single pass, no string duplication [8]
  • Hardware-friendly: Minimal allocations, good cache behavior
  • Code complexity: Adding rules requires threading lookahead logic into a nested loop
  • Error-prone pointer arithmetic: Handling nested quotes or ellipses requires care

4. Two-Pass Candidate Detection

This architecture (used by yasbd-lib [9]) flips the traditional logic. Instead of shielding text ahead of time, it aggressively identifies all potential boundaries first, then filters out false positives.

  • Pass 1: Find every position that could plausibly end a sentence: periods, question marks, exclamation points followed by whitespace, uppercase, or newline. Deliberately over-inclusive.
  • Pass 2: Evaluate each candidate against protection rules (abbreviation lists, decimal patterns, URL detection), discarding invalid targets. 
# Simplified example
def two_pass_detect(text):
    # Pass 1: find ALL potential boundary candidates
    candidates = [m.start() for m in re.finditer(r'(?<=[.!?])(?=\s+[A-Z])', text)]

    # Pass 2: filter out invalid points
    real_boundaries = []
    for pos in candidates:
        word_start = text.rfind(" ", 0, pos) + 1
        if text[word_start:pos].lower() not in ABBREVIATIONS:
            real_boundaries.append(pos)

    return real_boundaries
Enter fullscreen mode Exit fullscreen mode

Trade-offs:

  • Harder to miss a boundary: Over-inclusive by design [9]
  • Separates scanning from validation: Cleaner code organization
  • Synchronization risk: Pass 1 and Pass 2 must remain aligned
  • Requires careful candidate detection: Patterns must match what the filter can evaluate

Design Considerations

Choosing an architecture involves trade-offs across several dimensions.

Accuracy

All rule-based systems depend on the quality and completeness of their rule sets. The mask-then-split approach can achieve high accuracy on curated test sets, but its monolithic rule pipeline makes targeted fixes difficult. Two-pass systems isolate validation, making it easier to add or modify rules without affecting candidate detection [9].

Performance

Mask-then-split performs multiple passes and string allocations, which can become a bottleneck on large documents [4]. Pointer-based and two-pass approaches minimize allocations and are generally faster on modern hardware [8, 9].

Maintainability

State machines and pointer-based parsers embed logic in control flow, making rule additions invasive. Mask-then-split centralizes rules but creates interdependencies between transformations. Two-pass systems separate detection from filtering, simplifying rule management [9].

Span Tracking

If your pipeline needs character offsets into the original text (for NER training, RAG indexing, or translation alignment), consider how each architecture handles span reconstruction. Mask-then-split requires post-processing matching to map boundaries back to the original string [2]. Pointer-based and two-pass systems track positions directly, avoiding this step.

Multilingual Support

Rule-based SBD across languages requires per-language abbreviation lists, quotation rules, and punctuation conventions. Some architectures make this easier than others. PySBD supports 22 languages via modular language profiles [2]. yasbd-lib supports 39 languages using a declarative rule system that isolates language-specific data [9]. Two-pass systems with external language packs can add new languages without modifying core code [9].

Recommendations

There is no single "best" architecture—each has trade-offs suited to different constraints.

Constraint Recommended Architecture
Simple implementation, modest text sizes Mask → Split → Demask
Integration with existing tokenizers State machine
Minimal memory usage, large documents One-pass pointer-based
Ease of rule maintenance and extension Two-pass candidate detection
Accurate character span tracking Pointer-based or two-pass
Multiple languages Modular language profiles (any architecture)

For new projects that require:

  • High accuracy on diverse text
  • Easy rule maintenance without breaking the pipeline
  • Native character span tracking
  • Multilingual support

...a two-pass or pointer-based architecture with modular language profiles is often the most sustainable choice.

For existing pipelines that already use a mask-then-split library, the question is whether the maintenance burden and performance characteristics are acceptable for your use case.

References

[1] E. Stamatatos, N. Fakotakis, and G. Kokkinakis. "Automatic extraction of rules for sentence boundary disambiguation." Proceedings of the Workshop on Machine Learning in Human Language Technology, University of Patras, pp. 88-92.

[2] N. Sadvilkar and M. Neumann. "PySBD: Pragmatic Sentence Boundary Disambiguation." arXiv:2010.09657, 2020. https://arxiv.org/abs/2010.09657

[3] Pragmatic Segmenter (Ruby). https://github.com/diasks2/pragmatic_segmenter

[4] A. M. Kuchling. "Python Performance Tips." Python Documentation. https://docs.python.org/3/howto/performance.html

[5] spaCy Sentencizer. https://spacy.io/api/sentencizer

[6] F. Leitner. "syntok: Text tokenization and sentence segmentation." https://github.com/fnl/syntok

[7] M. Fowler. Refactoring: Improving the Design of Existing Code. Addison-Wesley, 1999.

[8] J. Dean and S. Ghemawat. "MapReduce: Simplified Data Processing on Large Clusters." OSDI 2004.

[9] yasbd-lib Documentation. https://github.com/speedyk-005/yasbd-lib/

Top comments (0)