This one took long due to the Analysis work I was doing for this post.There is a lot of work going on in the subject of Sentiment analysis so I decided to compare the accuracy of the products. Let's start with some basics...
NLP: Natural Language Processing
Natural Language Processing is a very interesting topic and a subject of debate when it comes to accuracy of the NLP.
Natural Language is very ambiguous as same sentences can have different meanings like
"I saw a man on a hill with a telescope. "
It seems like a simple statement until you begin to unpack the many alternate meanings:
- There’s a man on a hill, and I’m watching him with my telescope.
- There’s a man on a hill, who I’m seeing, and he has a telescope.
- There’s a man, and he’s on a hill that also has a telescope on it.
- I’m on a hill, and I saw a man using a telescope.
- There’s a man on a hill, and I’m seeing him with a telescope.
Sarcasm is that component of the language that is difficult for quite a few humans to understand.
Phrase: "Great job."
Normally means: You have created something of value here. Sarcastic way means: You have created nothing of value here. Also every day there are some new slangs added in the vocabulary like ROFL, LOL, NaMo, BrExit etc...
The presence of all these components in the language makes it much more difficult to process it.
Sentiment Analysis
It is a sub-domain of NLP which is basically a binary classification problem[To know more about ML and classification check my post [Machine Learning -A Solution or Problem]
The problem is to determine the polarity i.e the negativity or positivity of a sentence by processing the sentence and applying some algorithm to it.
One of the naive approach for calculating the polarity of the sentence is Bag of Words Technique.
In this technique, a sentence is simply broken into words and these words are then mapped to a dictionary which contains words and the polarity of the corresponding word.The total polarity of a sentence is calculated by summing up the individual words. An example of such dictionary is AFINN.
But this is pretty basic approach and works only for very simple sentences and not effective for complex language. There are many other algorithms like SVM, Maximum Entropy etc which calculate the polarity by form a relationship between the words.
Twitter Sentiment Analysis
Twitter is a very popular social network where information spreads like a fire and reaches millions of users within seconds. Twitter sentiment analysis is a subject in which the Twitter tweets are analysed for calculating polarities and then a social notion about a topic is estimated.
There are many products in the market that claim to provide market analysis based on the Twitter data.
A big question on this type of analysis is that if it is difficult to understand the natural language and even after too much research its accuracy in sentiment analysis is at most 85%, how difficult it is to classify a twitter tweet into positive or negative tweet by just 160 characters, also these tweets have lots of jargons and slangs.
To explore this I took a set of 100 tweets and ran sentiment analysis through three models
- IBM Watson
- CoreNLP by Stanford
- Microsoft Cognitive Service
- Manual Tagging
There was a maximum accuracy of 50% and all the models nearly performed equally well.
The results can be seen here.
One thing that was clear from this analysis is that
"Either we can not depend on Twitter's tweet data for Market Analysis and Predictions through Sentiment Analysis or we need a more sophisticated system to do the analysis."
Oldest comments (0)