Before 0.12.0, yasbd-lib was a promise. 39 languages, spaCy and pysbd integrations, a CLI, a streaming API, a benchmark suite. Every one of those language profiles took real time to build because we wanted them to survive actual text, not just the sentences we wrote to test them.
Reader, they did not survive actual text.
The fixes in this post come from a lot of people. Some opened a PR with the fix already in it. Some fixed a rule I wrote and got wrong. Names are credited where they did the work.
So we stopped adding languages. We froze the set at 39, locked the door (issue #198), and spent the next few months breaking everything we'd built. That's what this post is about.
Release: v1.0.0
If you're here for the pysbd comparison instead, that's a different post: yasbd-lib vs. pysbd: Two Philosophies of Sentence Boundary Detection.
Everyone else, let's get into it.
Why the language pack is locked
Every built-in language is a promise. You own its bugs, its weird abbreviations, its section markers, its edge cases, forever. Even if you never touch that language again. Even if you forget it exists. It's yours.
At 39 languages, that closet was full.
New languages go through external packs now. yasbd-auxlang covers Esperanto, Interlingua, Interlingue, and Ido. yasbd-union is an experiment in mixed-text segmentation where the language isn't known up front.
from yasbd import BoundaryDetector
detector = BoundaryDetector(
lang="eo",
external_lang_packs=["yasbd_auxlang"],
)
print(list(detector.segment("Saluton. Kiel vi fartas?")))
# ['Saluton.', 'Kiel vi fartas?']
The core stays at 39. It spends its time on bug fixes now, not on new coverage.
Is the freeze forever? No. If a language gets enough real demand, it can come back into the core. There's no fixed threshold for that yet, and I'd rather say so than pretend there's a process.
Everything below runs on that one decision. Frozen features, leftover work: correctness.
Lowering the Python floor
0.13.0 dropped the minimum Python from 3.11 to 3.10, because OpenMed wanted it. OpenMed supports 3.10. We supported 3.11. One of those was going to move, and it wasn't going to be them. That change is PR #212.
Same release, a cleanup pass in PR #203. _apply_cleaning_pipeline moved out of StreamCleaner.__next__, CLEANING_PIPELINE became DEFAULT_CLEANING_PIPELINE, the detect() offset math got simpler, TextSpan.__eq__ got shorter, and six rule files got their stray double spaces fixed. No user saw any of it. But the code was nicer to read, and we were about to freeze it, so that mattered.
The first round of boundary fixes
Three bugs in 0.13.0 set the pattern for everything after.
PR #207 fixed a double boundary on .\n, contributed by @MasRama. detect() was returning two offsets for the same boundary, one at the period and one at the newline. Reconstruct sentences from those offsets and you got an empty span in the middle. PySBD treats .\n as one boundary. So do we now.
list(detector.detect("Hello world.\nHow are you?"))
# [13, 25] <- one boundary at the period, not two
PR #211 fixed flattened lists. QUOTE_AND_PAREN_END_FINDER was firing outside its range. We wrapped it in an inner_range check and loosened the adjacency check so list markers at different distances count as the same list.
PR #204 fixed vertical lists. VERTICAL_LIST_START_FINDER only matched single-character markers and didn't anchor to line start, so 12. in a numbered list looked like a sentence boundary.
text = "Steps:\n10. Install the package\n11. Run the detector\n12. Read the output"
list(detector.segment(text))
# ['Steps:', '10. Install the package', '11. Run the detector', '12. Read the output']
# Each item is its own sentence; "10." is not a boundary mid-sentence
0.13.1 was a patch on the same finder. PR #214 caught a case where VERTICAL_LIST_START_FINDER matched any 1 to 4 character alphanumeric prefix as a list marker. Ordinary prose started losing boundaries. The fix restricts detection to pure digits and single-letter plus optional digit markers.
The hook, and why it exists
0.14.0 added a hook callback to BoundaryDetector in PR #234. Runs per paragraph, after the language rules apply. Gets a dict with text, lang, boundaries, paragraph_index. Can add or remove boundaries in place. HookError if it fails or produces invalid boundaries.
from yasbd import BoundaryDetector
def split_chinese_semicolons(ctx):
text = ctx["text"]
for i, ch in enumerate(text):
if ch == "；" and i not in ctx["boundaries"]:
ctx["boundaries"].append(i + 1)
return ctx
detector = BoundaryDetector(lang="zh", hook=split_chinese_semicolons)
Why does the hook exist? Because OpenMed was reaching into our internals with a monkey-patch called _split_yasbd_chinese_semicolons. That's a compliment in one sense and a design smell in another. Instead of fighting it, we made the hack a supported feature.
0.15.1 tightened the validation in PR #264. Type and bounds checks became single-pass guards with specific error messages, so a bad hook fails loudly instead of quietly corrupting the stream. @NataliaPerez08 then made sure offsets get deduplicated before sorting in PR #261, so boundaries stay unique.
This is what pre-1.0 is for. You can turn a hack into an API while you still have room to shape it.
Cleaning, hyphens, and line endings
0.14.0 also reworked the default cleaning pipeline. @HeaTTap added normalize_newlines in issue #232 to fold \r\n and \r into \n, and removed a destructive slash normalization step that had been eating legitimate slashes. The hyphenated word finder got expanded suffixes in PR #249.
from yasbd.utils.cleaner import StreamCleaner
# Wrap-around hyphenation now rejoins correctly
list(StreamCleaner("The state-of-the-\nart system works."))
# ['The state-of-the-art system works.']
Two fixes came with it. PR #231 stopped the cleaner from dropping a hyphen when a line broke at a real compound. state-of-the-\nart had been turning into state-of-theart, which is not a word in any language I know.
@ColumbusLabs fixed a related case in PR #255, preserving word boundaries across single line breaks so OCR text doesn't lose its spaces.
PR #236 fixed offsets on inputs with leading blank lines. detect() was skipping whitespace-only paragraphs before accumulating the offset, so if your text started with blank lines, every reported boundary was short by the length of those blanks.
text = "\n\nHello world. How are you?"
# Before: offsets were relative to "Hello", short by two newlines
list(detector.detect(text)) # [12, 25]
# After: offsets count from the true start
list(detector.detect(text)) # [14, 27]
PR #230 moved build_abbr_pattern into yasbd.utils.trie and renamed it build_optimized_pattern.
The fix that started with a wrong bug report
This part is a little embarrassing, but it's the reason the stress test exists.
In August I opened an issue on someone else's repo. abraham-jacob/scout had a workaround that protected abbreviations before sentence splitting. I looked at it and thought, "yasbd already handles U.S. and U.K.. Why is this thing here?"
So I filed an issue with a repro to prove it.
The library owner, Jacob Abraham, re-ran it. Confirmed my cases passed. Then pointed out that my repro didn't test anything, because both my examples had a lowercase word after the abbreviation:
"Full-time employees outside the U.S. receive full benefits."
# ^ lowercase
A boundary detector fires on period-whitespace-capital. Lowercase word after the period means the trigger never even gets a chance to fire. My test would have passed on a version with no abbreviation handling at all. I tested the wrong thing, wrote a careful issue about it, and was confidently wrong.
The cases that mattered were sitting in real job postings:
list(d.segment("This role requires U.S. Government security clearance."))
# Before 0.15.0: ['This role requires U.S.', 'Government security clearance.']
list(d.segment("This job is open to U.S. Persons only per export law."))
# Before 0.15.0: ['This job is open to U.S.', 'Persons only per export law.']
U.S. Persons is an export-control term. Always capitalized. U.S. Government shows up in every defense contractor job ad ever. Both are one sentence. yasbd was splitting them in two.
The reason: DOTTED_GEOPOL_ABBRVS in base.py feeds a heuristic in en.py that checks for a follow-up organizational noun. U.S. Department of Labor passed. U.S. Government and U.S. Persons didn't, because they weren't in the noun list.
I expanded ORG_PROPER_NOUNS in PR #257 and shipped it in 0.15.0. That closed issue #253.
I picked heuristics over blanket suppression on purpose. PySBD takes the simpler route: capitalized word after U.S. or U.K., don't split. That fixes the false split but also swallows real sentence boundaries:
"This policy was adopted in the U.S. Next week we will review it."
# Two sentences. pysbd keeps them as one.
The expanded noun list catches the false splits without eating the real ones. More work. Better result.
Jacob came back with round two. He audited the remaining abbreviations and found Inc., Corp., Ltd., Co., Jr., Sr., and etc. all doing the same thing:
list(d.segment("Acme Inc. USA is expanding its engineering team this quarter."))
# ['Acme Inc.', 'USA is expanding its engineering team this quarter.']
list(d.segment("John Doe Sr. VP of Engineering will be your hiring manager."))
# ['John Doe Sr.', 'VP of Engineering will be your hiring manager.']
The en.py heuristic doesn't cover corporate suffixes or name abbreviations, so the 0.15.0 fix didn't touch them. I shipped a second fix in 0.15.1 via PR #262, closing issue #260.
He didn't just catch my bad repro. He went through the rest of the list and handed me the next round of bugs. Two releases in this post exist because he took the time.
Three weeks later I opened issue #280. The whole point of it was to run that exact process on purpose, on all 39 profiles, instead of waiting for it to happen to me one repo at a time.
Making the heavy dependency optional
A user on r/yasbd_lib left a complaint that stuck with me. Longtime pysbd user, runs his systems entirely on RAM, said yasbd was forcing "more than 100MB" of dependencies and asked why it couldn't be as small and self-contained as pysbd.
I measured before I answered. Wheel was 95 KB. All 7 runtime deps: about 13.5 MB. The 100MB number didn't match unless dev extras had leaked into a production install.
But he was right about the thing that mattered, and the second look found something worse. py3langid, the package behind lang="auto", pulls in numpy. The import chain loaded it eagerly, so import yasbd dragged numpy in for everyone, including people who never touch auto-detection. On platforms without numpy wheels, it could break the install completely.
One user complaining about a number that was wrong, and underneath it was a real bug he hadn't even mentioned.
The fix shipped in 0.15.0. py3langid came out of the core deps in PR #256 and is now imported lazily inside classify_language().
pip install yasbd-lib # no numpy
pip install yasbd-lib py3langid # adds auto-detection
from yasbd import BoundaryDetector
# This works without py3langid installed
detector = BoundaryDetector(lang="en")
# This needs py3langid, installed separately
auto = BoundaryDetector(lang="auto")
That was the moment the library got serious about what it pulls in. Same question came up again later with loguru and ftfy. Same answer both times.
Closing the API
Between 0.15.1 and 0.16.2, the public surface got locked down.
0.16.0 is the big one. PR #270 removed the global _LANG_PACK_REGISTRY. BoundaryDetector now has its own registry.
# Before
from yasbd import register_lang_packs, clear_lang_packs
register_lang_packs(["yasbd_auxlang"])
detector = BoundaryDetector(lang="eo")
clear_lang_packs()
# After
detector = BoundaryDetector(
lang="eo",
external_lang_packs=["yasbd_auxlang"],
)
# No global state, no cleanup, two detectors can use different packs
register_lang_packs became load_external_lang_packs. clear_lang_packs got deleted. Two detectors no longer share global state, which is the right design, but it broke anyone using the old names. That's a change you can only make before 1.0, and we made it while we still could.
0.16.1 added a \b to the CORP_ENTITY_ABBRVS pattern in PR #271.
list(detector.segment("He works in tobacco. Co. is next."))
# ['He works in tobacco.', 'Co. is next.']
# "co" inside "tobacco" no longer matches as a corporate abbreviation
0.16.2 pinned retrie, beartype, and radicli, because upstream releases should not be able to break your install while you sleep.
The stress test
Issue #280 was the gate for v1.0.0. I wrote the 39 language rules. I also wrote the unit tests for those rules. Same person, same blind spots, so the tests couldn't catch what I'd gotten wrong. Untested assumptions slip through both.
This wasn't a theory. It had already happened to me three weeks earlier, on the scout repo.
So I ran the same process on purpose. Two weeks, all 39 profiles, real text through each one. News articles and papers and forum posts and chat logs. Every time a profile split something it shouldn't, I filed an issue. The list got long. Burmese double-comma boundaries. Amharic quotatives. Malayalam title abbreviations and section markers. And more.
Then contributors opened PRs to fix them.
@nightcityblade fixed the "for example" abbreviations in Hindi, Lithuanian, Malayalam, and Russian in PR #291. @YuEfSaEDU added the missing mev honorific to Dutch in PR #297, which Afrikaans inherits.
detector = BoundaryDetector(lang="af")
list(detector.segment("Mev. Jansen praat. Sy is hier."))
# ['Mev. Jansen praat.', 'Sy is hier.']
@be-student made tel. and fax. shared reference abbreviations in PR #284, added Portuguese aprox. in PR #282, and fixed Burmese double-comma handling in PR #286.
detector = BoundaryDetector(lang="pt")
list(detector.segment("O total, aprox. 500 reais."))
# ['O total, aprox. 500 reais.']
@sonalisrisivani switched NEWLINE_INSIDE_SENTENCE_FINDER to the Unicode lowercase property \p{Ll} in PR #276, so newlines before Cyrillic and other non-ASCII lowercase letters stop being treated as boundaries.
detector = BoundaryDetector(lang="ru")
list(detector.segment("Проф. Петров А.К. прибыл."))
# ['Проф. Петров А.К. прибыл.']
@HarshRajSinghania added the стор. and кв. reference abbreviations for Russian and Ukrainian in PR #308. That was his first contribution to the project. @Nagulanvelu fixed Lithuanian number-first section markers in PR #330. Also a first. @Chirudeva-Reddy fixed Burmese section markers and discourse particle boundaries in PR #329. Also a first. PR #306 and PR #307 kept Markdown headings with trailing numbers intact.
list(detector.segment("### 1. The Regex Breakdown\nSome text."))
# ['### 1. The Regex Breakdown', 'Some text.']
@DYNOSuprovo did the most of anyone in this batch. Amharic reporting words and converb forms in PR #332, optional whitespace before quotative particles, Swahili address and currency abbreviations in PR #310, German and Dutch unit abbreviations in PR #313, and a documentation example for sentence modification using boundary offsets in PR #312. That was the last cluster of fixes before the tag.
Burmese is my favorite of the batch. ၏ is sometimes a terminator and sometimes a possessive. ၊ is a sentence-ending comma. You can't know which without reading the sentence.
detector = BoundaryDetector(lang="my")
text = "သူက 'ဟုတ်ကဲ့' လို့ ပြောတယ်။ နောက်တစ်ခု။"
list(detector.segment(text))
# ["သူက 'ဟုတ်ကဲ့' လို့ ပြောတယ်။", 'နောက်တစ်ခု။']
Thirty-nine people have contributed to yasbd-lib since the project started, not just in this stretch. See CONTRIBUTORS.md for the full list.
There are almost certainly more bugs in profiles that got less attention and in edge cases nobody thought to try. 1.0.0 means the API is stable. It does not mean the rules are perfect.
Dropping what we didn't need
Three changes landed in this window, and two of them were removals.
PR #317 replaced loguru with a small stdlib logger in utils/logger.py. PR #325 dropped ftfy from the cleaner and swapped ftfy.fix_text for a lightweight _clean_mojibake step. PR #333 cleaned up the regex hot paths in the boundary engine and refreshed the benchmark to match.
The numbers from PR #325:
| Input Type | ftfy.fix_text |
_clean_mojibake |
|---|---|---|
| Plain sentence (39 chars) | 57.7 µs | 2.8 µs |
Mojibake Café résumé
|
163.6 µs | 2.7 µs |
HTML entities &
|
150.1 µs | 22.4 µs |
| Long text (1350 chars) | 264.7 µs | 18.1 µs |
Here's what the table doesn't say. _clean_mojibake does less than ftfy did. It handles the common cases: cp1252 and latin-1 misreads, HTML entity unescaping, non-breaking space normalization. ftfy handled a wider range of encodings and a pile of edge cases the new step doesn't touch. If your text has exotic encoding damage, ftfy still repairs things the new step won't.
That was a choice. Most of the text that flows through this library comes from web scrapes, PDFs, and chat logs. Those produce a small set of recurring encoding problems, and _clean_mojibake covers them. The exotic cases were rare enough that carrying the dependency on every install wasn't worth it. If you do hit one, install ftfy and pass it as an extra step:
from ftfy import fix_text
from yasbd.utils.cleaner import StreamCleaner
cleaner = StreamCleaner(text)
The other half of the tradeoff is speed. When cleaning runs, the new step is faster everywhere. The mojibake case drops the most because that's where ftfy did the most work. HTML entities drop the least because unescaping is real work either way.
Where this shows up: the PySBD adapter with cleaning on. Segmenter defaults to clean=False, same as pysbd, so you opt in:
from yasbd.pysbd_adapter import Segmenter
# clean=False by default, matching pysbd.
# Pass clean=True to run the StreamCleaner step.
seg = Segmenter("en", clean=True)
list(seg.segment(text))
When cleaning is on, every paragraph pays the cost. That cost used to be ftfy.fix_text on every paragraph. Now it's stdlib. On a document with a lot of paragraphs, and text with a lot of mojibake, it adds up.
Same defaults, less overhead, one fewer dependency to install. Full benchmark methodology is in PR #325.
What stable means
v1.0.0 means the public API is settled. It does not mean every language profile is perfect.
from yasbd import BoundaryDetector, register_spacy_component
# The shapes that stay the same for the 1.x line:
detector = BoundaryDetector(lang="en")
list(detector.detect("Hello. World.")) # [6, 13]
list(detector.segment("Hello. World.")) # ['Hello.', 'World.']
# StreamCleaner, ParagraphStream, the pysbd adapter, and the spaCy
# component are frozen interfaces now.
The 39 built-in languages stay frozen for now. New coverage enters through lang packs first, and only the profiles that prove themselves in real use have a path back into the core.
The bugs will keep coming. Real text is too strange for that to stop.
But we got the practice in early. That's what the stretch from 0.12.0 to now was for.
Top comments (0)