Vo Viet Hoang

Posted on Aug 26

Building a Client-Side N-gram Utility for Text Structure and Phrase Audit

#webdev #showdev #javascript #vibecoding

Hey DEV community! 👋

When writing technical documentation, user guides, or long-form informational articles, maintaining a clear and engaging reading style is highly important. However, as writers, we often fall into repetitive phrasing habits without realizing it.

Traditional word counters only track isolated, single words. To evaluate multi-word phrases and understand the flow of our writing, we need a different approach. This is where an N-gram analysis becomes highly useful.

To provide a safe and private solution for content editors, I built a lightweight, entirely client-side N-gram Analyzer.

In this post, we will explore the technical implementation of this utility, how to handle text segmentation in JavaScript, and why local browser processing is a reliable choice for data privacy.

What is an N-gram?

In computational linguistics and text processing, an N-gram is a contiguous sequence of $n$ items (usually words) from a given sample of text.

A Unigram represents single words ($n=1$).
A Bigram represents two-word phrases ($n=2$).
A Trigram represents three-word phrases ($n=3$).
A 4-gram represents four-word phrases ($n=4$).

Analyzing these combinations helps developers and content creators identify repetitive phrases, evaluate vocabulary diversity, and check if the thematic distribution of a document aligns with its target focus.

The Client-Side Approach: Privacy and Data Isolation

Many online text tools process user inputs on backend servers. This setup introduces a significant privacy risk if you are analyzing sensitive internal documentation, unpublished drafts, or proprietary code comments.

By executing the lexical parsing entirely within the user's browser, we keep the processing local. The text never travels across the network, and there are no external database logs. The local device handles the entire operation.

Implementing the N-gram Extraction in JavaScript

Let's look at the core logic. To build an N-gram extractor, the utility must perform three key tasks:

Clean the Input: Remove standard punctuation marks to ensure consistent word matching.
Segment into Words: Split the text into individual words while supporting standard character sets.
Build the Map: Use a sliding window to slice the word array into sequences of size $N$ and count their occurrences.

Here is the clean JavaScript implementation:

/**
 * Extracts N-grams from a raw text string.
 * @param {string} text - The input prose to analyze
 * @param {number} n - The length of the phrase (1 for Unigram, 2 for Bigram, etc.)
 * @param {number} minOccurrence - Filter out phrases appearing less than this threshold
 * @returns {Array} Sorted array of [phrase, count] pairs
 */
function extractNgrams(text, n, minOccurrence = 2) {
    if (!text) return [];

    // Clean text: remove standard punctuation and split into lowercase words
    // The regular expression supports accented characters and common diacritics
    const words = text.toLowerCase()
        .replace(/[.,\/#!$%\^&\*;:{}=\-_`~()]/g, "")
        .match(/\b[\w\u00C0-\u024F\u1E00-\u1EFF]+\b/g) || [];

    const totalWords = words.length;
    if (totalWords < n) return [];

    const ngramsMap = {};

    // Sliding window algorithm
    for (let i = 0; i <= totalWords - n; i++) {
        const gram = words.slice(i, i + n).join(" ");
        ngramsMap[gram] = (ngramsMap[gram] || 0) + 1;
    }

    // Filter by minimum occurrence and sort by frequency in descending order
    return Object.entries(ngramsMap)
        .filter(([_, count]) => count >= minOccurrence)
        .sort((a, b) => b[1] - a[1]);
}

Code Explanation:

Text Normalization: We convert the text to lowercase so that capitalized words at the beginning of sentences match their lowercase counterparts in the body text.
Unicode Compatibility: The character range \u00C0-\u1EFF in the regular expression ensures that accented characters in various languages are processed correctly, avoiding broken words.
The Sliding Window: The loop moves forward one word at a time. It slices a segment of length n, joins the words with a single space, and increments the frequency counter in our map object.

Designing the User Experience

To make the utility practical and accessible, the frontend layout is built with standard, responsive grid components:

Flexible Text Area: A roomy input box where users can paste their text blocks.
Adjustable Parameters: Dropdown selectors to change the $N$ parameter (from Unigram to 4-gram) and a numeric input to set the minimum occurrence threshold.
Structured Results Table: A dynamic display showing the rank of each phrase, the raw text sequence, the exact frequency count, and the percentage representation relative to the entire document.

This design enables content editors to quickly identify if specific phrases appear too frequently, allowing them to vary their vocabulary before publishing.

Try the Live Utility

I have integrated this private text analyzer directly into my site as part of a collection of utility scripts. If you need a rapid, secure way to audit your writing patterns or inspect document structures without any backend tracking, feel free to use it:

👉 Live Link: Online N-gram Analyzer

Let's Connect!

How do you approach text analysis and readability reviews in your development workflow? Do you write local script setups, or do you rely on text-processing utilities in your terminal?

Feel free to share your thoughts and suggestions in the comments section below. Let's keep learning together! 🚀

DEV Community