Token Jaccard Similarity in TypeScript: Simple Text Comparison

Hey TypeScript devs! 👋

If you need a fast, lightweight way to measure how similar two pieces of text are (duplicate detection, content recommendations, search result grouping, or data cleaning), Token Jaccard Similarity is an excellent choice.

It’s simple to understand, works great in both browser and Node.js, and requires zero dependencies.

What is Jaccard Similarity?

The Jaccard index measures similarity between two sets:

$$J(A, B) = \frac{|A \cap B|}{|A \cup B|}$$

• Intersection = tokens present in both texts
• Union = all unique tokens from either text

Result ranges from 0 (completely different) to 1 (identical).

Token Jaccard = Jaccard applied to words

We tokenize the text into words, convert them into Sets (removing duplicates and order), then apply the formula.

Quick Example

Text A:

the quick brown fox jumps over the lazy dog

Text B:

a quick brown fox jumps over a lazy dog

Unique tokens:

• Text A: the, quick, brown, fox, jumps, over, lazy, dog
• Text B: a, quick, brown, fox, jumps, over, lazy, dog

Intersection: 7 tokens

Union: 9 tokens

Jaccard Similarity ≈ 0.78 (78% similar)

TypeScript Implementation

Here’s a clean, fully typed implementation:

function tokenize(text: string): Set<string> {
  // Matches words (alphanumeric), lowercased
  const matches = text.toLowerCase().match(/\b\w+\b/g);
  return new Set(matches ?? []);
}

function jaccardSimilarity(text1: string, text2: string): number {
  const set1 = tokenize(text1);
  const set2 = tokenize(text2);

  if (set1.size === 0 && set2.size === 0) {
    return 1;
  }

  // Calculate intersection size
  let intersectionSize = 0;
  for (const token of set1) {
    if (set2.has(token)) {
      intersectionSize++;
    }
  }

  const unionSize = set1.size + set2.size - intersectionSize;

  return intersectionSize / unionSize;
}

// ======================
// Example usage
// ======================

const textA = "the quick brown fox jumps over the lazy dog";
const textB = "a quick brown fox jumps over a lazy dog";

const score = jaccardSimilarity(textA, textB);
console.log(`Jaccard Similarity: ${score.toFixed(4)}`); 
// → Jaccard Similarity: 0.7778

Another Example

const textC = "Building a recommendation system using TypeScript and machine learning";
const textD = "How to create a recommendation engine with TypeScript and ML";

console.log(jaccardSimilarity(textC, textD).toFixed(2)); 
// → 0.50

Real-World Use Cases

Use Case	Why It Works Well in TypeScript	Suggested Threshold
Duplicate content detection	Extremely fast even on large datasets	> 0.85
Article / blog recommendations	Good for topic similarity	0.4 – 0.65
Dataset deduplication	Lightweight preprocessing step	> 0.90
Search result clustering	Quick fuzzy grouping	> 0.70
Basic plagiarism detection	Simple and explainable baseline	> 0.60

Limitations & Quick Improvements

Token Jaccard is great but has trade-offs:

• Ignores word order and semantics
• Sensitive to common words (“the”, “a”, “is”)

Easy Upgrades

1. Remove stop words before tokenizing
2. Use n-grams (word or character shingles) instead of single words
3. Combine with other metrics (e.g. cosine similarity on embeddings)

But for most practical needs, plain Token Jaccard in TypeScript is already very effective and blazing fast.

When to Use It

• You want something simple and dependency-free
• You’re working in TypeScript / JavaScript (frontend, backend, or full-stack)
• You need explainable results
• You’re building a quick prototype or data pipeline

Conclusion

Token Jaccard Similarity remains one of the most practical tools for text comparison. With this TypeScript version, you can drop it into any project instantly — browser, Node.js, Deno, or even Edge functions.

Would you like me to show:

• A version with stop-word removal?
• N-gram (shingle) based Jaccard?
• How to use it efficiently at scale?

Just let me know in the comments!

Comments

[Frank]

How does Token Jaccard Similarity handle stop words and punctuation, I'd love to hear your thoughts on optimizing this for larger texts.

[Christopher S. Aondona]

Token Jaccard Similarity usually handles stop words and punctuation through preprocessing, which involves normalizing text, removing punctuation, and optionally filtering stop words before token comparison. For larger texts, optimize by using efficient token sets, hashing, and techniques like MinHash for faster similarity estimation at scale.

[Christopher S. Aondona]

Are you interested in learning more about AI memory?