DEV Community

OnePassApps
OnePassApps

Posted on

How the Text Studio Counts Words, Characters, Sentences and Bytes: 41 Probes

I work on OnePassApps, and we build the browser studios. This one is the Text Studio on app.onepassapps.com - and the part worth measuring is the row of counters that sits under the input box.

Most "word counter" widgets are a box with one number in it. This one has five numbers and a limit counter, and they do not all count the same thing. I typed 41 probe strings into the live input and checked candidate rules for each number against every probe, and all six rules matched every time.

What the row actually shows

Under the "01 Input text" step there is a live stats row with five labels:

  • Words
  • Characters
  • Sentences
  • Lines
  • UTF-8 bytes

Above it, next to the step number, the input limit counter reads 0 / 2,000,000 and ticks up as you type. The panel under the input says "Nothing leaves this tab." and the bottom of the studio says "Processing stays in this browser."

The counters are live: they update as you type, with no button press. I confirmed that by typing "One two three." with real key events into the field and reading Words 3, Characters 14, Sentences 1, Lines 1, UTF-8 bytes 14 back off the page.

The six rules, and how I pinned them

For every probe string I computed candidate formulas inside the same page (the browser's own Intl.Segmenter, TextEncoder, a regular expression and a spread over the string) and compared each candidate with the number the panel was showing. All 41 probes matched all six rules - 246 individual comparisons, no mismatches.

Probe Words Characters Sentences Lines UTF-8 bytes limit counter
Hello world. 2 12 1 1 12 12
Hello world 2 11 0 1 11 11
Hello. World. 2 13 2 1 13 13
Wait... what? 2 13 2 1 13 13
Dr. Smith went home. 4 20 2 1 20 20
Pi is 3.14 exactly. 4 19 1 1 19 19
Line one + newline + Line two 4 17 0 2 17 17
Line one + newline + Line two + newline 4 18 0 3 18 18
Café naïve 2 10 0 1 12 10
well-known 2 10 0 1 10 10
Hi + emoji + there 2 10 0 1 13 11
你好世界 2 4 0 1 12 4
https://example.com/some/path 5 29 0 1 29 29
(three spaces) 0 3 0 1 3 3
empty input 0 0 0 0 0 0

Characters counts Unicode code points, not bytes and not UTF-16 code units. Every string I tried agreed with counting the code points of the string.

UTF-8 bytes is the byte count you would get by encoding the same string as UTF-8. This is where the two diverge: Café naïve is 10 characters and 12 bytes, and the four CJK characters 你好世界 are 4 characters and 12 bytes.

Words is not a whitespace split. On all 41 probes it matched the word-like segments produced by the browser's built-in word segmenter, and that difference is visible in ordinary text: well-known counts as 2 words where a whitespace split would say 1, https://example.com/some/path counts as 5 where a whitespace split would say 1, and 你好世界 counts as 2.

Sentences counts terminator marks, not sentences. A period, question mark or exclamation mark that is followed by whitespace or by the end of the text ends a sentence, and the count is the number of those marks. So Hello world with no terminator reads 0 sentences, not 1. Dr. Smith went home. reads 2, because the abbreviation period is followed by a space. Pi is 3.14 exactly. reads 1, because the point in 3.14 is followed by a digit and does not count.

Lines is the number of line breaks plus one. A trailing newline adds a line, so two lines of text with a final newline reads 3. An empty input reads 0 lines, and an input holding just a newline reads 2.

The limit counter counts UTF-16 code units - the length of the string as JavaScript sees it - which is why it disagrees with the Characters row on exactly one kind of input. A string with a character outside the Basic Multilingual Plane, such as an emoji, costs two code units but only one character: the row reading Hi + emoji + there showed Characters 10, UTF-8 bytes 13 and the limit counter at 11. That counter is the one that runs against the 2,000,000 input limit.

Why the rules matter for a counter you might paste into

If you paste an article into that field and read the numbers, the sentence count is the one that will surprise you. A paragraph with no full stop at the end reads 0 sentences. An abbreviation in the middle of a sentence splits it in two. A decimal point does not. The word count behaves differently from a whitespace split in the way you would expect from a real segmenter: hyphenated words, URLs and CJK runs are broken into words rather than left as one blob.

And if you are checking that a file fits a limit somewhere else, the UTF-8 bytes row is the number to read, not the Characters row. A few emoji and some accented characters move the byte count away from the character count, and the input limit counter moves with the code units instead.

Honest limits of this measurement

  • One browser, one build, one day, signed out.
  • These rules are inferred from exact agreement across 41 probes. I did not read the application's source, and I have no access to it, so a later build could count differently.
  • The counters read the input. I did not check the result stage for counters of its own.
  • The stats row is present on the Clean, Case and Base64 tabs and absent on the Diff tab. I checked those four tabs of the eight.
  • I did not test the 2,000,000 limit by pasting two million characters into the field.
  • I did not compare these numbers against another word counter.

The Text Studio is part of the standard membership, and Standard is free right now: $0 / month, 100,000 credits every 30 days, No end date set, No card required, with no automatic paid subscription.

Guides for every studio: the app guides

Disclosure: I work on OnePassApps - we build these studios.

Top comments (0)