Using a scikit wrapper to easily tune a pre-trained BERT model for NLP Brand Sentiment Analysis of Social Media Data
aAuthor: Dylan Dey
This project it available on github here: link
The Author can be reached at the following email: ddey2985@gmail.com
Classification Metric Understanding
Confusion Matrix Description
A true positive in the current context would be when the model correctly identifies a tweet with positive sentiment as positive. A true negative would be when the model correctly identifies a tweet with negative sentiment as containing negative sentiment. Both are important and both can be described by the overall accuracy of the model.
True negatives are really at the heart of the model, as this is the situation in which Apple would have a call to action. An appropriately identified tweet with negative sentiment can be properly examined using some simple NLP techniques to get a better understanding at what is upsetting customers involved with our brand and competitor's brands. Bigrams, quadgrams, and other word frequency analysis can help Apple to address brand concerns.
True positives are also important. Word frequency analysis can be used to summarize what consumers think Apple is doing right and also what consumers like about Apple's competitors.
There will always be some error involved in creating a predictive model. The model will incorrectly identify positive tweets as negative and vice versa. That means the error in any classification model in this context can be described by ratios of true positives or negatives vs false positives or negatives.
A false positive would occur when the model incorrectly identifies a tweet containing negative sentiment as a tweet that contains positive sentiment. Given the context of the business model, this would mean more truly negative sentiment will be left out of analyzing key word pairs for negative tweets. This could be interpreted as loss in analytical ability for what we care about most given the buisness problem: making informed decisions from information directly from consumers in the form of social media text. Minimizing false positives is very important.
False negatives are also important to consider. A false negative would occur when the model incorrectly identifies a tweet that contains positive sentiment as one that contains negative sentiment. Given the context of the business problem, this would mean extra noise added to the data when trying to isolate for negative sentiment of brand/product.
In summary, overall accuracy of the model and a reduction of both false negatives and false positives are the most important metrics to consider when developing a the twitter sentiment analysis model.
All functions used to preprocess twitter data, such as removing noise from text and tokenizing, as well as the functions for creating confusion plots to quickly assess performance are shown below.
#list of all functions for modeling
#and processing
#force lowercase of text data
def lower_case_text(text_series):
text_series = text_series.apply(lambda x: str.lower(x))
return text_series
#remove URL links from text
def strip_links(text):
link_regex = re.compile('((https?):((\/\/)|(\\\\))+([\w\d:#@%\/;$()~_?\+-=\\\.&](#!)?)*)|{link}/gm')
links = re.findall(link_regex, text)
for link in links:
text = text.replace(link[0], ', ')
return text
#remove '@' and '#' symbols from text
def strip_all_entities(text):
entity_prefixes = ['@','#']
for separator in string.punctuation:
if separator not in entity_prefixes:
text = text.replace(separator,' ')
words = []
for word in text.split():
word = word.strip()
if word:
if word[0] not in entity_prefixes:
words.append(word)
return ' '.join(words)
#tokenize text and remove stopwords
def process_text(text):
tokenizer = TweetTokenizer()
stopwords_list = stopwords.words('english') + list(string.punctuation)
stopwords_list += ["''", '""', '...', '``']
my_stop = ["#sxsw",
"sxsw",
"sxswi",
"#sxswi's",
"#sxswi",
"southbysouthwest",
"rt",
"tweet",
"tweet's",
"twitter",
"austin",
"#austin",
"link",
"1/2",
"southby",
"south",
"texas",
"@mention",
"ï",
"ï",
"½ï",
"¿",
"½",
"link",
"via",
"mention",
"quot",
"amp",
"austin"
]
stopwords_list += my_stop
tokens = tokenizer.tokenize(text)
stopwords_removed = [token for token in tokens if token not in stopwords_list]
return stopwords_removed
#master preprocessing function
def Master_Pre_Vectorization(text_series):
text_series = lower_case_text(text_series)
text_series = text_series.apply(strip_links).apply(strip_all_entities)
text_series = text_series.apply(unidecode.unidecode).apply(html.unescape)
text_series =text_series.apply(process_text)
lemmatizer = WordNetLemmatizer()
text_series = text_series.apply(lambda x: [lemmatizer.lemmatize(word) for word in x])
return text_series.str.join(' ').copy()
#function for intepreting results of models
#takes in a pipeline and training data
#and prints cross_validation scores
#and average of scores
def cross_validation(pipeline, X_train, y_train):
scores = cross_val_score(pipeline, X_train, y_train)
agg_score = np.mean(scores)
print(f'{pipeline.steps[1][1]}: Average cross validation score is {agg_score}.')
#function to fit pipeline
#and return subplots
#that show normalized and
#regular confusion matrices
#to easily intepret results
def plot_confusion_matrices(pipe):
pipe.fit(X_train, y_train)
y_true = y_test
y_pred = pipe.predict(X_test)
matrix_norm = confusion_matrix(y_true, y_pred, normalize='true')
matrix = confusion_matrix(y_true, y_pred)
fig, (ax1, ax2) = plt.subplots(ncols = 2,figsize=(10, 5))
sns.heatmap(matrix_norm,
annot=True,
fmt='.2%',
cmap='YlGn',
xticklabels=['Pos_predicted', 'Neg_predicted'],
yticklabels=['Positive Tweet', 'Negative_Tweet'],
ax=ax1)
sns.heatmap(matrix,
annot=True,
cmap='YlGn',
fmt='d',
xticklabels=['Pos_predicted', 'Neg_predicted'],
yticklabels=['Positive Tweet', 'Negative_Tweet'],
ax=ax2)
plt.show();
#loads a fitted model from memory
#returns confusion matrix and
#returns normalized confusion matrix
#calculated using given test data
def confusion_matrix_bert_plots(model_path, X_test, y_test):
model = load_model(model_path)
y_pred = model.predict(X_test)
matrix_norm = confusion_matrix(y_test, y_pred, normalize='true')
matrix = confusion_matrix(y_test, y_pred)
fig, (ax1, ax2) = plt.subplots(ncols = 2,figsize=(10, 5))
sns.heatmap(matrix_norm,
annot=True,
fmt='.2%',
cmap='YlGn',
xticklabels=['Pos_predicted', 'Neg_predicted'],
yticklabels=['Positive Tweet', 'Negative_Tweet'],
ax=ax1)
sns.heatmap(matrix,
annot=True,
cmap='YlGn',
fmt='d',
xticklabels=['Pos_predicted', 'Neg_predicted'],
yticklabels=['Positive Tweet', 'Negative_Tweet'],
ax=ax2)
plt.show();
Class Imbalance of Dataset
The twitter data used for this project was collected from multiple sources from CrowdFlower. The project will only focus on binary sentiment (positive or negative). The total amount of tweets and associated class balances are show below. This distribution is further broken down by brand in the chart below the graphs.
Apple Positive vs Negative Tweet Counts
positive 0.654194
negative 0.345806
+++++++++++++++++++++++
positive 2028
negative 1072
+++++++++++++++++++++++++++++++++++++++++++++++++++
Google Positive vs Negative Tweet Counts
positive 740
negative 136
+++++++++++++++++++++++
positive 0.844749
negative 0.155251
For comparison, I trained four different supervised learning classifiers using term frequency–inverse document frequency(TF-IDF) vectorized preprocessed tweet data. While the vectorization will not be needed for the BERT classifier, it is needed for these supervised classifiers.
TfidfVectorize sklearn documentation
Balanced Random Forest Classifier Documentation
Multinomial Naive Bayes Base Model Performance
Random Forest Classifier Base Model Performance
Balanced Random Forest Classifier Base Model Performance
XGBoosted Random Forest Classifier Base Model Performance
Now that supervised learning models have been built, trained, and tuned without any pre-training, our focus will now turn to transfer learning using Bidirectional Encoder Representations from Transformers(BERT), developed by Google. BERT is a transformer-based machine learning technique for natural language processing pre-training. BERTBASE models are pre-trained from unlabeled data extracted from the BooksCorpus with 800M words and English Wikipedia with 2,500M words.
Click Here for more from Wikipedia
Sckit-learn wrapper provided by Charles Nainan. GitHub of Scikit Learn BERT wrapper.
This scikit-learn wrapper is used to finetune Google's BERT model and is built on the huggingface pytorch port.
The BERT classifier is now ready to be fit and trained on data in the same way you would any sklearn model.
See the code block below for a quick example.
bert_1 = BertClassifier(do_lower_case=True,
train_batch_size=32,
max_seq_length=50
)
bert_1.fit(X_train, y_train)
y_pred = bert_1.predict(X_test)
Four models were trained and stored into memory. See the code bock below for the chosen parameters in every model.
"""
The first model was fitted as seen commeted out below
after some trial and error to determine an appropriate
max_seq_length given my computer's capibilities.
"""
# bert_1 = BertClassifier(do_lower_case=True,
# train_batch_size=32,
# max_seq_length=50
# )
"""
My second model contains 2 hidden layers with 600 neurons.
It only passes over the corpus one time when learning.
It trains fast and gives impressive results.
"""
# bert_2 = BertClassifier(do_lower_case=True,
# train_batch_size=32,
# max_seq_length=50,
# num_mlp_hiddens=500,
# num_mlp_layers=2,
# epochs=1
# )
"""
My third bert model has 600 neurons still but
only one hidden layer. However, the model
passes over the corpus 4 times in total
while learning.
"""
# bert_3 = BertClassifier(do_lower_case=True,
# train_batch_size=32,
# max_seq_length=50,
# num_mlp_hiddens=600,
# num_mlp_layers=1,
# epochs=4
# )
"""
My fourth bert model has 750 neurons and
two hidden layers. The corpus also gets
transversed four times in total while
learning.
"""
# bert_4 = BertClassifier(do_lower_case=True,
# train_batch_size=32,
# max_seq_length=50,
# num_mlp_hiddens=750,
# num_mlp_layers=2,
# epochs=4
# )
Bert 1 Results
Bert 2 Results
Bert 3 Results
Bert 4 Results
As you can see, all of my BERT models trained on a relatively small amount of data achieved much better results than any of the other classifiers. The BertClassifier with 1 hidden layer, 600 neurons, and 4 epochs performed the best, predicted over 93% of positive tweets correctly and 80% of negative tweets correctly on hold out test data.
bert_3 = BertClassifier(do_lower_case=True,
train_batch_size=32,
max_seq_length=50,
num_mlp_hiddens=600,
num_mlp_layers=1,
epochs=4
)
That concludes this example of using a pre-trained BERT model for Twitter sentiment analysis on a small set of data. If you have any questions or comments please feel free to contact me. Thank you.
Author: Dylan Dey
Email: Ddey2985@gmail.com
github: https://github.com/ddey117/Product_Twitter_Sentiment_Classification
Top comments (0)