DEV Community: Vladimir Vasilev

Detecting attempts of mass influencing via social networks with NLP. Part 2

Vladimir Vasilev — Wed, 29 Jun 2022 14:54:06 +0000

In Part 1 of this article, I built and compared two classifiers to detect trolls on Twitter. You can check it out.

Now, time has come to look more deeply into the datasets to find some patterns using exploratory data analysis and topic modelling.

EDA

To do just that, I first created a word cloud of the most common words, which you can see below.

So, “butthis” and “bestUSAtoday” are the most frequently used words, which seemingly makes no sense. However, butthis.com is actually one of the most famous websites within the Twitter account bestUSAtoday considered to be related to the IRA agency. The account itself was suspended by Twitter for violation of the Twitter Rules. Moreover, the butthis.com domain no longer exists as well, so I could only look at the screenshots of the site and, from what I saw, all its content was political. This word was mentioned 350 times in 2288 tweets, which makes this external resource popular among trolls.
Other most common words were ”isis” (ISIS, i.e. Islamic State of Iraq and the Levant), ”police”, ”potus” (President of the United States), ”news” and words related to the main competitors of the 2016 US Presidential Elections Hillary Clinton and Donald Trump.

Topic modelling

The next step, topic modelling, showed me the most common topics in the tweets under study. For this purpose, I used LDA, which requires a bag-of-words representation of the tweets as its input.

As part of this research, I compared Gensim LDA with Scikit Learn LDA, and it turned out that Scikit Learn does not provide a convenient coherence calculation model, which could allow me to quickly obtain the coherence measure given a certain number of topics. That’s why I chose Gensim LDA as the model with a broader range of the required features.

Speaking of the coherence measure, it shows the level of correlation between words in a topic. The higher the coherence is the more sense in a topic we will observe. In my experiments, I was looking for the so-called coherence k-value, which represents the peak of the rapid growth of coherence, and found out that the optimal number of the topics would be 7.

*As for the results of the topic modelling: *

Topic: 0 is about the above-mentioned troll resources ”bestUSAtoday”, butthis.com and other online resources, such as Newsmax.com or TheHuffingtonPost.com, which are private online news and opinion websites in the US. Topic: 5 concerns Donald Trump with words like ”Iran”, ”deal”, ”irandeal”,”potus”, ”irannucleardeal” or ”trump2016”. This topic covers the year of the nuclear deal between Iran and a group of six countries led by POTUS Donald Trump. Another interesting Topic: 4 is dedicated to refugees and migration and includes such words as “attacks”, “could” or “kill”. In addition, there is Topic: 2, which is about the immigration caused by the Syrian war in the Middle East and about how candidates for the position of the President were going to deal with it. Topic: 6 is fully dedicated to the 2016 elections and the sources of propaganda.

Conclusions and further work

So, two classifiers with very high overall accuracy, precision, recall and F-1 score have been built and tested on several features. The experimental results have shown that the tweet text as a feature gives better accuracy than hashtags. The models were trained on large amounts of data and, thus, can be used as a correct solution for detecting attempts at mass influencing. The exploratory analysis and topic modelling allowed me to delve deeper into the actual goals of the trolling. All this makes it possible to conclude that the ability to affect internal affairs of another state exists but, probably, depends on the amount of the troll army used and how they are used.
This study in two parts is initial and can be furthered with deep learning and analysis of visual components (such as images and videos attached to the tweets). To continue with the research, I’m going to build a classifier that will take into account such aspects as the user name, the user picture, kind of account activities (number of retweets and likes), number of tweets per unit of time and some deeper parameters like the type of followers or the sentiment of the tweets.
The only concern related to more advanced analysis and deep learning is that troll accounts will soon be automated and converted into bot accounts. This means that we will soon need to extract features of bots instead of trolls. This might be quite a complicated thing because bots usually have many similar features, but not all of them represent trolls.

Detecting attempts of mass influencing via social networks with NLP. Part 1

Vladimir Vasilev — Wed, 29 Jun 2022 14:47:34 +0000

During the last decades, the world’s population has been developing as an information society, which means that information started to play a substantial end-to-end role in all life aspects and processes. In view of the growing demand for a free flow of information, social networks have become a force to be reckoned with. The ways of war-waging have also changed: instead of conventional weapons, governments now use political warfare, including fake news, a type of propaganda aimed at deliberate disinformation or hoaxes. And the lack of content control mechanisms makes it easy to spread any information as long as people believe in it.

Based on this premise, I’ve decided to experiment with different NLP approaches and build a classifier that could be used to detect either bots or fake content generated by trolls on Twitter in order to influence people.
In this first part of the article, I will cover the data collection process, preprocessing, feature extraction, classification itself and the evaluation of the models’ performance. In Part 2, I will dive deeper into the troll problem, conduct exploratory analysis to find patterns in the trolls’ behaviour and define the topics that seemed of great interest to them back in 2016.

Features for analysis

From all possible data to use (like hashtags, account language, tweet text, URLs, external links or references, tweet date and time), I settled upon English tweet text, Russian tweet text and hashtags. Tweet text is the main feature for analysis because it contains almost all essential characteristics that are typical for trolling activities in general, such as abuse, rudeness, external resources references, provocations and bullying. Hashtags were chosen as another source of textual information as they represent the central message of a tweet in one or two words.

Among all the other features, I also considered using a tweet’s date and time metadata because it seemed pretty handy according to various experiments. However, it turned out that there were no patterns in the data on the troll tweets I obtained.

Data collection

Having done an extensive research of the state-of-the-art techniques dealing with the troll problem, I proceeded with data collection. Data sources vary: you can either use datasets provided by Twitter itself, research and data platforms or by companies such as kaggle.com or data.world. Besides, there are some data sets freely available on github.com provided by enthusiasts. However, after some rumination, I’ve decided to use the Twitter data to avoid untrustworthy sources and irrelevant data especially taking into account that Twitter, by that time, announced that they were making publicly available archives of tweets and media that they assumed were created from potentially state-backed information operations.
For the “troll” data, I was especially interested in two datasets: the one that was potentially related to tweets from Russia and the other one dedicated to the tweets of Internet Research Agency (IRA). This agency was considered responsible for a potential state-backed campaign aimed at influencing people in the USA and, thus, affecting Presidential Elections. In particular, there were 2288 and 765,248 values in each dataset, respectively.
For the “non-troll” (user) data, I used Twitter-provided API and downloaded almost 100,000 tweets on different topics, including books, news and politics. Tweets on such topics as news and politics are supposed to be close to the topics discussed by troll accounts, and quotations from books containing conversations on the relevant topics were added to create a negative dataset with diversified discussion. I also used regular conversational tweets from verified users.
In general, the more similar positive and negative data are the more relevant results of defining ”trolls” or ”non-trolls” we receive.

Preprocessing and feature extraction

Speaking of feature extraction, a classical Bag of Words approach is still the most popular solution for this kind of work. At the same time, the Word Embeddings approach is considered more sophisticated. Usually, the classification of tweets requires a good lexicon representation. So, I compared them using TfidfVectorizer (chosen over CountVectorizer as the latter cannot process large amounts of data) for the bag-of-words model and Glove for the word embeddings one.
In terms of preprocessing, for efficient classification, in this case I needed to conduct tokenisation, normalisation, noise removal, stopword removal, lowercasing and lemmatisation (only for the word embeddings).
For lowercasing, my datasets included a lot of abbreviations, such as “US” or “U.S.”, which, if converted to “us”, could affect the experiment, so I had to think carefully before using preprocessing incorporated in the libraries providing different bag-of-words models, and conduct various experiments before picking the right model.
It is also not always recommended to use lemmatisation or stemming because it could theoretically lead to a change of initial meaning of the unigram which, in its turn, might substitute the overall result, especially taking into account that I was going to visualise different patterns of the initial tweet text. However, in reality, it turned out that lemmatisation actually works very well with the word embeddings model and leads to higher accuracy when it comes to further classification.
Another issue to ponder on was stopword removal. At first glance, it seems obvious that such words as “the”, “and” or “a” should be removed but they can be informative for future predictions when classifying the style of writing or personality (which might be the case here). In the end, however, I removed them for the word embeddings model because the advantages of such removal outweighed the possible disadvantages, which was noticeable in the final classification results.

Classification and evaluation

Based on the literature review for the classification part of the project, it was decided to use Support Vector Machine and Multinomial Naive Bayes Classifiers. They are well-recognised classifiers among data scientists as some of the best solutions for tweet data.
For the SVM, I was interested in C-Support Vector Classification and Linear Support Vector Classification classifiers for comparison. It turned out that the SVC classifier cannot process vast amounts of data for a reasonable amount of time while LinearSVC was pretty fast. Higher evaluation scores were also received with LinearSVC.
The Multinomial Naive Bayes classifier is considered one of the most efficient solutions for text classification with such discrete features as word counts. Thus, it works perfectly with the Bag of Words model. The multinomial distribution typically requires integer feature counts. Therefore, it was decided to include this classifier in the analyses to conduct binary classification.

SVM is the classifier that is usually trained on data belonging to one of two classes. The classifier infers the properties of ‘good’ cases and from these properties can predict which examples are unlike the common examples. Binary SVM is trained on data with both classes. The classifier predicts classes by finding the best hyperplane that separates all data points of one class of those from another class.
The Naive Bayes classifier is very efficient in supervised learning settings. Although there is not a single algorithm for training such classifiers, there is a universal principle: all Naive Bayes classifiers assume that all the features are independent of each other, based on these features classes might be predicted.

According to the results of my experiments, for all three cases (English tweet text, Russian tweet text, and hashtags), among the feature extraction models, the most significant contribution to classification was made by TFIDFVectorizer, which implied the classical Bag of Words approach. Moreover, it was the winning model for both LinearSVC and MultinomialNB classifiers, which made each of them predict more accurately than LinearSVC with all possible Glove models.

Below you can find a table with metrics I used for analysing the results of the classification where TP is the number of true positive predictions; TN is the number of true negative predictions; FP is the number of false positive predictions; and FN is the number of false negative predictions.

According to my results, the accuracy measure of LinearSVC classification performance was better than the Multinomial Naive Bayes classifier in all three cases. You can see the evaluation report and the confusion matrix for the English tweet text classification below.

Continued with EDA, topic modelling, and conclusions in Part 2.

How to build a pipeline for document recognition in a week but troubleshoot FineReader and NER issues for a year

Vladimir Vasilev — Wed, 22 Jun 2022 13:51:32 +0000

In this article, I would like to describe how we’ve tackled the document recognition issue mostly with the help of OCR, BERT and Yandex’s Natasha, with the end goal being extraction of entities for reconciliation followed by a payment. Abby FineReader is a well-known optical character recognition application developed by ABBYY. BERT is an open-source transformer-based ML framework used for a variety of NLP tasks pre-trained by Google on 3.3B words. Natasha is currently a set of open-source NLP tools for the Russian language that solves such tasks as tokenization, sentence segmentation, word embedding, morphology tagging, lemmatisation, phrase normalization, syntax parsing, NER tagging. The quality of every task is similar or better than current state-of-the-art for the Russian language on news articles.

Just imagine countless textual documents even a medium-sized organisation deals with on a daily basis, let alone huge corporations. Our company, for example, collaborates with hundreds of suppliers, contractors and other counterparties, which implies thousands of contracts. For instance, the estimated number of legal documents to be processed in 2022 has been over 70,000, each of them consisting of 30 pages on average. During the lifecycle of a contract, it is usually updated with 3 to 5 additional agreements. On top of this, a contract is accompanied by various source documents describing transactions. And in the PDF format, too.

Previously, the processing duty befell our service centre’s employees who checked whether payment details in a bill match those in the contract and then sent the document to the Accounting Department where an accountant double-checked everything. This is quite a long journey to a payment, right?

The human factor added to the problem: the task of entering information into an ERP system is not your dream job and is quite mundane, which significantly affects the quality of the resulting data, not to mention that the entire process is relatively slow: according to our calculations, it takes about 3.5 minutes for an employee to process one contract. In addition, employees extract entities from documents only partially, for specific purposes, while all entities contain valuable information that we could use for other projects.

Here comes our document recognition task...

The contemporary tech space offers multiple intelligent document recognition solutions but none of them quite befitted our purposes (aside from, probably, ABBYY FineReader) because we needed a universal solution.

Typically, for optical character recognition (OCR) tasks, flexible templates put on top of the document structure are used. If the structure is the same, the information is retrieved with high quality. The same process is applicable to tables. It may seem like recognising tabular data is a simple problem because structure is virtually the definition of a table. But there are some buts here. For example, different types and formatting of the cells, or wrong association between a cell and the text it contains, in the OCR results.

With this problem in mind and having carefully considered all pros and cons of creating our own product, we’ve come up with a solution that works for all kinds of tables. And here is how: a flexible template first recognises the borders of the table, with three JSON files containing the border coordinates, a textual layer and metadata on the recognition results generated, and then recalculates them, eliminating inaccuracies in the coordinates. This way we can have the coordinates of all elements of the table, and it is quite easy to proceed from there with the usual NLP tasks. In some respects, this solution is unique and is the heart of our proprietary AI platform. We are currently in the process of migrating from ABBYY FR to a more sophisticated OCR solution developed on site.

As for the architecture specifics, the system includes the source and target systems, which is our ERP system, PDF documents to be processed, and the AI platform itself, with an integration layer between SAP and the platform. First, the system recognises the document structure, then classifies the documents and pastes together contracts with additional agreements, after which relevant entities are finally extracted.

One of the problems that we faced at the first step was some loss of information because some of our entities were handwritten and, besides, had stamps upon them, which was problematic for ABBYY FR to process. This situation, however, was not common as it occurred only in 3% of the cases. Another issue was data corruption due to low quality of scanned documents, some of which dated back to the 1980s.

Then comes the second stage where we recognise a document structure using fundamental models to segment the document into sections, classify the segments and pages, as well as to recognise individual clauses and subclauses in legal texts. All entity extraction models heavily depend on this step because they look up only certain sections of the contract to ensure better quality of entity extraction. The same goes for the model joining documents together because we need first to recognise the amendment type (these being removals, additions or replacements) and then to amend the contract accordingly.
Our next AI model classifies incoming documents into groups, such as contracts vs. additional agreements, and further, into more specific groups like signed vs. unsigned contracts, and so on. This step eliminates the need for manual selection of the document to be recognised and provides for downloading documents en masse.

The contract joining model is followed by the entity extraction models. Speaking of the types of entities, we saw that dates and numbers of contracts and signatures, as well as amounts showed good recognition quality of more than 80%. Addresses, names and positions, however, required further refinement. The most problematic ones, with the quality of less than 70%, were signing dates, contract start and end dates, and the subject of the agreement, and these are the top-priority entities for labelling.

In our workflow, the models extract up to 44 entities from every contract and additional agreement and up to 20 entities from source documents. All the entities from the updated contract are then reconciled with those from the relevant source documents. And if there is a match, an auto payment can be made.

Seeing as 100 people out of 149 employees of our service centre are engaged in reconciliation activities, the solution we have offered will greatly optimise the headcount in the centre as well as speed up the reconciliation process (1 minute vs. 3.5 minutes).

As for the quality of the document processing as measured with standard metrics such as precision, recall and F-measure, our models have shown the accuracy of 90% for the majority of the entities vs. expected 60%.

Another important implication of this project is that we have received valuable historical data that can be further used in other projects and for decision-making.

We have been continuously improving this project and trying to form a common understanding of the labelling, development and validation processes with the business customer. Seeing that the entity extraction quality is directly related to the labelling quantity/quality and that only the business members of the team can expertly assess what entities must be extracted, we have needed their help with additional labelling because some fundamental models (like section or page classification, or (sub)clause extraction models) require retraining with more data. The business customer’s engagement in the results validation and motivation of the team responsible for the related work on the ERP system and development of the integration layer are also important.