Bag Of Words

#ai

Bag of Words (BoW) is a foundational technique in text processing, where text is transformed into numerical vectors based on word presence and frequency. It is a simple yet powerful method for converting text data into a format that machine learning models can understand.

Why the Name "Bag of Words"?

The term "Bag of Words" comes from the idea that the model treats text like a "bag" of words:
- It only cares about the presence of words (do they exist?) and their frequency (how often they appear).
- Like items in a physical bag, the words are placed in without concern for their order or arrangement.

Core Purpose

Transform text (words or sentences) into numeric representations that machine learning models can understand.

What It Solves

Transforms Text into Numeric Vectors: Each unique word in the text is represented as a feature (column) in a vector.
Encodes Text into Fixed-Length Representations: Each sentence is converted into a vector of word counts, ensuring consistent vector size.

Popular Applications of Bag of Words (BoW)

1) Text Classification:
Used with algorithms like Naive Bayes for spam detection or sentiment analysis. Each document is transformed into a Bag of words vector, and the model learns word probabilities for each class (e.g., spam or not spam).

2) Document Similarity (Cosine Similarity):
Bag of words vectors allow for measuring similarity between documents using Cosine Similarity, which is useful in search engines and recommendation systems.

3) Topic Modeling (Latent Dirichlet Allocation [LDA]):
Bag of words provides the word distribution used by LDA to discover hidden topics in a collection of documents.

Advantages & Limitations

Advantages
1) Easy to Understand: Quick to implement without complex logic.
2) Efficient for Small Datasets: Performs well with basic text processing tasks.
3) Compatible with Basic Models: Works seamlessly with algorithms like Naive Bayes and Logistic Regression.

Limitations
1) No Context Awareness: Ignores word order and sentence structure.
2) High Dimensionality: Large vocabulary results in sparse, high-dimensional vectors.
3) Lacks Semantic Understanding: Words are treated independently, without meaning.

Core Logic

Vocabulary Creation: Extracts unique words from the initial text.
Text Vectorization: Converts new text into a vector using the fixed vocabulary.
Reusability: The fixed vocabulary ensures consistency across multiple texts.

Sample Python Implementation

import nltk
nltk.download('punkt')
import string

# Step 1: Build Vocabulary from Initial Text
def build_vocabulary(text):
    dataset = nltk.sent_tokenize(text)
    vocabulary = set()

    for sentence in dataset:
        for word in nltk.word_tokenize(sentence):
            word = word.lower().strip(string.punctuation)
            if word:
                vocabulary.add(word)

    return sorted(list(vocabulary)) 

# Step 2: Convert New Text to Bag of Words Vector Using the Fixed Vocabulary
def text_to_bag_of_words(text, vocabulary):
    dataset = nltk.sent_tokenize(text)
    word2count = dict.fromkeys(vocabulary, 0) 

    for sentence in dataset:
        for word in nltk.word_tokenize(sentence):
            word = word.lower().strip(string.punctuation)
            if word in word2count:
                word2count[word] += 1

    vector = [str(word2count[word]) for word in vocabulary]
    return vector

# Initial Text (Training Text)
initial_text = "Python is great for data science. Coding is fun!"
vocabulary = build_vocabulary(initial_text)

print("Vocabulary (Fixed):", vocabulary)

# Using the Fixed Vocabulary to represent New Text
new_text = "Python is amazing. Data science is evolving."
vector = text_to_bag_of_words(new_text, vocabulary)

print("Bag of Words Vector for New Text:")
print(f"[{', '.join(vector)}]")

Output


Vocabulary (Fixed): ['coding', 'data', 'for', 'fun', 'great', 'is', 'python', 'science']

Bag of Words Vector for New Text:
[0, 1, 0, 0, 0, 2, 1, 1]

Conclusion

Bag of Words (BoW) is a foundational text processing technique known for its simplicity and transparency. Despite its limitations, understanding BoW is crucial because it builds the foundation for grasping more advanced methods in Natural Language Processing.

DEV Community