Text Analytics Techniques
Text analytics is a process where an artificial intelligence (AI) algorithm, running on a computer, evaluates these same attributes in text, to determine specific insights. A person will typically rely on their own experiences and knowledge to achieve the insights. A computer must be provided with similar knowledge to be able to perform the task. There are some commonly used techniques that can be used to build software to analyze text, including:
Statistical analysis of terms used in the text. For example, removing common "stop words" (words like "the" or "a", which reveal little semantic information about the text), and performing frequency analysis of the remaining words (counting how often each word appears) can provide clues about the main subject of the text.
Extending frequency analysis to multi-term phrases, commonly known as N-grams (a two-word phrase is a bi-gram, a three-word phrase is a tri-gram, and so on).
Applying stemming or lemmatization algorithms to normalize words before counting them - for example, so that words like "power", "powered", and "powerful" are interpreted as being the same word.
Applying linguistic structure rules to analyze sentences - for example, breaking down sentences into tree-like structures such as a noun phrase, which itself contains nouns, verbs, adjectives, and so on.
Encoding words or terms as numeric features that can be used to train a machine learning model. For example, to classify a text document based on the terms it contains. This technique is often used to perform sentiment analysis, in which a document is classified as positive or negative.
Creating vectorized models that capture semantic relationships between words by assigning them to locations in n-dimensional space. This modeling technique might, for example, assign values to the words "flower" and "plant" that locate them close to one another, while "skateboard" might be given a value that positions it much further away.
While these techniques can be used to great effect, programming them can be complex. In Microsoft Azure, the Language cognitive service can help simplify application development by using pre-trained models that can:
Determine the language of a document or text (for example, French or English).
Perform sentiment analysis on text to determine a positive or negative sentiment.
Extract key phrases from text that might indicate its main talking points.
Identify and categorize entities in the text. Entities can be people, places, organizations, or even everyday items such as dates, times, quantities, and so on.
In this module, you'll explore some of these capabilities and gain an understanding of how you might apply them to applications such as:
A social media feed analyzer to detect sentiment around a political campaign or a product in market.
A document search application that extracts key phrases to help summarize the main subject matter of documents in a catalog.
A tool to extract brand information or company names from documents or other text for identification purposes.
These examples are just a small sample of the many areas that the Language service can help with text analytics.
Top comments (0)