Classifying Social Media Texts into Multiple Hate-Based Categories for Indian languages

By:
K Chaitanya Balaji Reddy - SE23MAID007
M Batul Abbas - SE23MAID010
N Ananda Varaprasada Babu - SE23MAID011
K Harsha - SE23MAID006

Introduction

Hate speech is widespread on platforms like Twitter, Facebook, comment sections, blogs, and biased online publications. Existing profanity filters often miss subtle attacks or contextually hateful speech that uses seemingly harmless words. This project aims to develop a tool to detect hate speech in Hindi tweets, categorizing them as “weakly hateful,” “strongly hateful,” or “no hate” based on the level of hate content. Our model uses a rule-based approach, leveraging sentiment analysis and subjectivity analysis to assess the polarity of sentiment expressions. Objective sentences are removed from the corpus, and a hate speech classifier is built using a lexicon extracted from the corpus. The hate speech detector’s effectiveness is tested in real-world web discourse scenarios.

The Corpus

Political and social issues often trigger strong emotions, leading to rudeness and hate speech in discussions, especially on platforms like Twitter. Our corpus consists of tweets mostly of a political or social nature, pre-segregated into various hatefulness categories using a CSV file. The categories are:

• Non-hostile
• Fake
• Defamation
• Offensive
• Hate

These categories are not mutually exclusive; a tweet can belong to multiple categories simultaneously. For example, a tweet marked as fake could also be defamatory and offensive. Non-hostile tweets do not fall into any other categories as they are devoid of malintent.

The Algorithm

The algorithm is divided into three main steps:

1. Subjectivity Analysis
2. Building a Hate Speech Lexicon
3. Identifying Theme-Based Nouns

Subjectivity Analysis

Subjectivity analysis is a crucial step in our algorithm as it helps to filter out objective tweets that are unlikely to contain hate speech. Objective tweets typically present facts or information without any emotional bias, which makes them less likely to be hateful. By focusing on subjective tweets, which are more likely to express opinions and emotions, we can more accurately identify potential hate speech.

The Role of Subjectivity Analysis

In the context of hate speech detection, subjectivity analysis serves several important purposes:

1. Reduction of Noise: The primary function of subjectivity analysis is to reduce the noise in the dataset. Objective tweets, such as those reporting news or sharing information, do not contribute to the hate speech problem. By removing these tweets from the analysis, we can focus on the tweets that are more likely to contain hateful content, thereby increasing the efficiency and accuracy of our model.
2. Enhanced Precision: By filtering out objective tweets, we narrow down the dataset to those tweets that are more likely to express subjective opinions and emotions. This enhances the precision of our hate speech detection as the remaining tweets are more likely to contain language that reflects personal biases, prejudices, and potential hate speech.
3. Efficient Use of Resources: Processing a large dataset can be computationally intensive and time-consuming. By eliminating objective tweets, we can allocate our computational resources more effectively to analyze subjective tweets. This allows us to build a more focused and efficient model for detecting hate speech.
4. Contextual Relevance: Subjective tweets often contain context-specific language and expressions that are crucial for identifying hate speech. By concentrating on these tweets, we can better understand the context in which certain words or phrases are used, which is essential for accurately detecting hate speech that may be veiled or context-dependent.

Implementation with HindiWordNet & Stanza

To implement subjectivity analysis, we use HindiWordNet, a lexical database specifically designed for the Hindi language. HindiWordNet assigns positive and negative scores to each word based on its sentiment. The process involves the following steps:

1. Scoring Words: Each word in the tweet is assigned a sentiment score from HindiWordNet. Positive scores indicate a positive sentiment, while negative scores indicate a negative sentiment.
2. Aggregating Scores: The scores of all words in a tweet are aggregated to calculate a total sentiment score for the tweet.
3. Threshold for Subjectivity: Tweets with a total score above 1.0 or below -0.5 are considered sufficiently subjective for further analysis. The threshold values are determined based on empirical observations, where tweets with higher positive or negative scores are more likely to express strong opinions or emotions.
4. Filtering: Tweets that do not meet the subjectivity threshold are filtered out, leaving a subset of tweets that are more likely to contain subjective content and, consequently, potential hate speech.

The Stanza library, developed by the Stanford NLP Group, provides comprehensive support for Hindi language processing, making it an excellent tool for tasks like tokenization, part-of-speech tagging, and dependency parsing in hate speech detection projects. Its pre-trained models ensure high accuracy and efficiency in extracting linguistic features crucial for identifying and analyzing hate speech in Hindi.

Importance of Subjectivity Analysis

The need for subjectivity analysis in hate speech detection cannot be overstated. Without this step, the model would need to process a large number of objective tweets that do not contribute to the problem of hate speech, thereby diluting the effectiveness of the analysis. By focusing on subjective tweets, we can more accurately identify hate speech, improve the model’s performance, and ensure that our computational resources are used efficiently.

In conclusion, subjectivity analysis is a vital component of our hate speech detection algorithm. It allows us to filter out irrelevant content, focus on emotionally charged and opinionated tweets, and improve the precision and efficiency of our hate speech detection efforts.

Building a Hate Speech Lexicon

A critical component of our hate speech detection algorithm involves constructing a comprehensive lexicon of words and phrases that signify hate speech. This lexicon is built in multiple stages, using various linguistic resources and techniques to ensure its robustness and relevance.

Role of SUBJCLUE Lexicon

The first step in building our hate speech lexicon is to leverage the SUBJCLUE lexicon, which contains words annotated with sentiment scores. Words from the SUBJCLUE lexicon are categorized based on their sentiment scores:

• Strongly Negative Words: Words with a total score less than -0.25 are classified as strongly negative. These words are likely to carry strong negative sentiments and are essential for identifying intense hate speech.
• Weakly Negative Words: Words with scores greater than -0.25 but still negative are considered weakly negative. These words, while not as intense, still contribute to the overall negative sentiment and potential hate content in tweets.

This initial categorization helps in detecting the negative polarity of words, forming the foundation for our hate speech lexicon.

Expansion with SYNSET

The next crucial step involves expanding this initial lexicon using SYNSET, which plays a pivotal role in building a more comprehensive and interconnected web of lexicons.
What is SYNSET?

SYNSET (short for Synonym Set) refers to a set of synonyms that share a common meaning. In linguistic databases like WordNet or its regional variants like HindiWordNet, a SYNSET groups words that can be used interchangeably in various contexts. This semantic relationship is harnessed to enhance our lexicon’s coverage and accuracy.

Role of SYNSET in Lexicon Building

1. Identifying Seed Verbs: Initially, we manually select a list of seed verbs from the dataset that are believed to carry hateful connotations. These verbs are chosen based on their frequent usage in hateful tweets and their potential to express hate.
2. Expanding with Synonyms: Using SYNSET, we identify all possible synonyms of these seed verbs. This step significantly broadens the scope of our lexicon by including words that, while not present in the initial dataset, share the same hateful connotations. For example, if “attack” is a seed verb, SYNSET helps us include synonyms like “assault,” “bash,” “blast,” etc.
3. Forming a Hate-Verbs Lexicon: The expanded list of synonyms is then compiled into a final hate-verbs lexicon. This lexicon is more exhaustive and covers a wider range of hateful expressions, ensuring that our hate speech detection model can recognize various forms of hate speech, even if they use different words to convey the same sentiment.
4. Interconnected Lexicons: By leveraging SYNSET, we create a web of lexicons where each word is interconnected through its synonyms. This web-like structure enhances the model’s ability to detect hate speech by understanding the semantic relationships between words. It allows the model to capture nuanced expressions of hate that might otherwise be missed if only a limited set of words were considered.

Benefits of Using SYNSET

• Comprehensive Coverage: SYNSET ensures that the lexicon covers a broad spectrum of words with similar meanings, making the detection process more robust.
• Contextual Understanding: By including synonyms, the model gains a better understanding of the context in which certain words are used, improving the accuracy of hate speech detection.
• Adaptive Lexicon: The lexicon becomes adaptive and dynamic, capable of evolving as new synonyms and expressions of hate speech emerge over time.

In summary, SYNSET plays a crucial role in expanding the hate speech lexicon, creating a comprehensive and interconnected web of lexicons. This expansion enhances the model’s ability to detect hate speech accurately by incorporating a wide range of synonymous expressions, thereby improving the overall effectiveness of the hate speech detection algorithm.

Theme-Based Nouns

To accurately detect hate speech, it is essential to identify and analyze not only verbs and adjectives but also the nouns that often appear in hateful contexts. This step involves a detailed examination of noun phrases (NPs) within the corpus to pinpoint those frequently used in hateful tweets. By doing so, we can create a targeted list of nouns that enhance the precision of our hate speech detection model.

Importance of Analyzing Noun Phrases

Noun phrases are crucial in understanding the context and targets of hate speech. They often include the subjects and objects of sentences, which can reveal who or what is being targeted by hateful language. Analyzing noun phrases allows us to:

1. Identify Targets of Hate Speech: Noun phrases often include names of individuals, groups, or entities that are the targets of hate speech. Recognizing these targets helps in understanding the focus of the hate content.
2. Contextual Relevance: By examining the noun phrases, we can ensure that the identified hate speech is relevant to the specific topics being discussed. This relevance is vital for accurately detecting and categorizing hate speech.
3. Enhance Lexicon with Context-Specific Terms: Adding frequently used noun phrases to our lexicon ensures that the hate speech detection model captures context-specific hate speech that might otherwise be overlooked.

Process of Analyzing Noun Phrases

1. Segregating the Corpus: The corpus is divided into individual noun phrases. This involves using natural language processing techniques to parse sentences and extract noun phrases, which typically consist of a noun and its modifiers.
2. Frequency Analysis: Once the noun phrases are extracted, we conduct a frequency analysis to identify the most commonly used noun phrases in tweets marked as hateful. This step helps in recognizing the key terms that frequently appear in hate speech.
3. Building the Noun Lexicon: The recurrent nouns and noun phrases identified from the frequency analysis are compiled into a text file. This list forms a part of our hate speech lexicon, focusing specifically on nouns relevant to the discussion topics.
4. Ensuring Relevance: The noun phrases added to the lexicon are reviewed to ensure they are pertinent to the types of hate speech being analyzed. This step ensures that our model remains focused on the relevant context and does not include irrelevant or overly generic terms.

Benefits of Including Theme-Based Nouns

1. Improved Detection Accuracy: Including theme-based nouns in the lexicon improves the accuracy of the hate speech detection model by capturing context-specific language.
2. Context Awareness: The model becomes more aware of the context in which certain nouns are used, allowing for more precise identification of hate speech.
3. Enhanced Coverage: By identifying and including frequently used noun phrases, the lexicon covers a broader range of hate speech expressions, making the detection process more comprehensive.
4. Focus on Relevant Content: The focus on noun phrases ensures that the detection model targets relevant content, reducing false positives and increasing the precision of hate speech identification.

In summary, the process of analyzing noun phrases and incorporating frequently used nouns into the lexicon is a crucial step in enhancing the accuracy and relevance of our hate speech detection model. By focusing on context-specific terms, we ensure that the model can accurately detect and categorize hate speech in various contexts, thereby improving its overall effectiveness.

Final Testing

The lexicons are tested on the corpus to evaluate their impact on hate speech detection accuracy. The algorithm’s components are tested individually to assess each lexicon’s contribution. The following criteria are used to classify tweets:

• Strongly Hateful: Tweets with two or more strongly negative words, or a combination of strongly negative words with hate-verbs or theme-based nouns.
• Weakly Hateful: Tweets with one strongly negative word or a combination of weakly negative words with theme-based nouns or hate-verbs.
• Non-Hateful: Tweets that do not meet the above criteria.

The results are compared with the manually defined objective results using precision, recall, and F-score metrics.

As you can see, the hate-verbs alone do not make too much of a difference to the accuracy of our model, but adding the themed-noun lexicon improves it relatively significantly.

Clearly, using subjectivity clues makes a very significant improvement to the accuracy of our hate-speech detector (nearly 7% increase in the F-score), which shows the importance of checking for contextual nouns while searching for and detecting hate speech. Another noteworthy factor is how the Recall is much higher than the precision; much higher, in fact, than the Recall value of the original project we referred to even though the precision we get is relatively lower. We argue this is due to our algorithm being efficient with ensuring that the hate-speeches are tagged correctly, as are the innocuous tweets. Thus, it leaves few false negatives (i.e. misses) behind. There are, however, relatively more false positives which takes away from precision. We argue that this is caused by our objective criterion (as described above) not being particularly objective at separating weak hate from strong hate because of the misalignment between criteria used by the corpus to tag hate speeches, and the criteria we ourselves use to segregate them. However, our algorithm compares favorably to the original in terms of segregating non-hate tweets from hateful tweets (about 66% precision in segregating non-hate) because of the criteria being more objective.

Conclusion

The detection of hate speech in widely spoken languages like Hindi is increasingly important in today's digital era. Our model employs a rule-based approach to identify lexical patterns, allowing us to detect and quantify hate speech with reasonable precision. A key component of our algorithm is the analysis of subjectivity, which significantly enhances the accuracy of hate speech detection. Additionally, we categorize the detected hate speech to provide a more detailed understanding of the content, which helps prioritize our actions if needed.