DEV Community: J Mungai

GETTING STARTED WITH SENTIMENT ANALYSIS

J Mungai — Mon, 27 Mar 2023 20:45:43 +0000

Sentiment analysis is a Natural Language Processing technique used in determining the emotional tone or attitude behind a piece of text. This is also known as opinion mining.
For example, an organization's digital space has opinions form their clients. It is important for the organization to se these opinions to get insights about their products and services. To effectively analyze this data, the organization gathers all the opinions in one place and applies sentiment analysis to it, since going through all the data manually is almost next to impossible.
Use cases for sentiment analysis include:

Social media monitoring for brand management
Product/ service analysis
Stock price prediction
Customer feedback analysis

Several libraries are available in python for sentiment analysis. They include:

1. Natural Language Toolkit (NLTK): NLTK is popular for natural language processing in python. It provides a variety of tools for text analysis, including pre-trained sentiment analysis models like VADER.
2. TextBlob: This is a library for text processing and sentiment analysis. It provides a simple API for sentiment analysis an also includes features like part-of-speech tagging and noun phrase extraction.
3. SpaCy: SpaCy provides tools for tokenization, part-of-speech tagging and dependency parsing, as well as pre-trained models for sentiment analysis
4. Scikit Learn: It is a popular machine learning library in python. it provides a variety of tools for text analysis including algorithms for sentiment analysis, like Naïve Bayes and Support Vector Machines.
5.Tensorflow and Keras: These provide tools for building and training deep learning models which can be used for sentiment analysis tasks.

There are several approaches to sentiment analysis, each with its own strengths and weaknesses. The best approach depends on specific needs of the application. They include:

1. Role-Based Approach: This approach relies on manually crafted rules or lexicons to identify sentiment in text. It is based on the assumption that certain words and phrases are inherently positive or negative. For example, the word "happy" is generally considered positive, while the word "sad" is considered negative.
This approach is often relatively simple and transparent but can be limited by specificity and comprehensiveness of the lexicons or rules used.
2. Machine-Learning Approach: Involves training a model on a labelled dataset of text and sentiment. The model uses this training to predict the sentiment of new text. Machine learning approach can be highly accurate but requires a large amount of labelled data to train the model effectively. Common algorithms include Naive Bayes, Support Vector Machines and neural networks.
3. Hybrid Approach: Combines both rule-base and machine-learning approaches.
For example: A hybrid approach might use a lexicon of sentiment-bearing words to identify sentiment in text and then use a machine-learning model to fine tune the sentiment analysis based on context. This approach provides the best of both worlds, but can be complex and difficult to implement.
4. Deep-Learning Approach: Involves training a neural network to learn the representation of text and sentiment. These models can learn complex relationships between words and are often highly accurate. They however require a large amount of labelled data and can be computationally expensive to train.
5. Lexicon-Based Approach: This is a rule-based approach that uses a pre-built sentiment lexicon to determine the sentiment of a text. A sentiment lexicon is a collection of words or phrases with their corresponding polarity, i.e. negative or positive. The sentiment of a text is determined by counting the number of positive and negative words in the text. This approach is fast and easy to implement, but can be limited by the size and quality of the sentiment lexicon used.

Below is an example of sentiment analysis using the twitter dataset on Kaggle:

Load the dataset into a Pandas DataFrame

import pandas as pd
df = pd.read_csv('entity_sentiment_twitter.csv')

Print the first 5 rows of the DataFrame

print(df.head())

                                                text     entity  sentiment
0  I'm excited to share my new course on @kaggle ...     Kaggle   Positive
1  @elonmusk thanks for the Tesla. Can't believe ...  @elonmusk   Positive
2     @SpotifyCares I need help with my account pls.   SpotifyC   Negative
3  Had a great experience with @Apple customer s...      Apple   Positive
4            My favorite game is @PlayHearthstone\n  Hearth...   Positive

import re

def clean_text(text):
    # Remove URLs
    text = re.sub(r'http\S+', '', text)

    # Remove hashtags and mentions
    text = re.sub(r'#\w+', '', text)
    text = re.sub(r'@\w+', '', text)

    # Remove special characters and punctuation
    text = re.sub(r'[^\w\s]', '', text)

    # Convert to lowercase
    text = text.lower()

    return text

# Apply the clean_text function to the 'text' column of the DataFrame
df['clean_text'] = df['text'].apply(clean_text)

from textblob import TextBlob

def get_sentiment(text):
    # Create a TextBlob object from the text
    blob = TextBlob(text)

    # Get the polarity score (-1 to 1)
    polarity = blob.sentiment.polarity

    # Classify the sentiment as positive, negative, or neutral based on the polarity score
    if polarity > 0:
        sentiment = 'Positive'
    elif polarity < 0:
        sentiment = 'Negative'
    else:
        sentiment = 'Neutral'

    return sentiment

# Apply the get_sentiment function to the 'clean_text' column of the DataFrame
df['predicted_sentiment'] = df['clean_text'].apply(get_sentiment)

# Calculate the accuracy of the sentiment analysis
accuracy = (df['sentiment'] == df['predicted_sentiment']).mean()

print(f'Accuracy: {accuracy:.2%}')

Accuracy: 69.58%

EXPLORATORY DATA ANALYSIS ULTIMATE GUIDE

J Mungai — Tue, 28 Feb 2023 12:47:43 +0000

Exploratory data analysis (EDA) is an iterative process that involves visualizing and summarizing data to gain insights and inform further analysis, which makes it a crucial step in any data analysis process.
EDA helps to look at data before making any assumptions. It can help identify errors as well as identify and better understand patterns within the data, detect outliers and find interesting relations.
EDA can aid in answering questions about standard deviations, categorical variables and confidence intervals. Once EDA is complete and insights are drawn, its features can be used for more sophisticated data analysis on modelling.

Exploratory Data Analysis Tools

Python:: This is a programming language used for data analysis and visualization. It provides libraries for EDA such as NumPy, Pandas, MatplotLib and Seaborn among others.

2.Tableau: This is a visualization tool that allows the analyst to create interactive visualizations and dashboards. It is user friendly and has a wide range of visualization options.

3.Excel: This is a popular spreadsheet tool that can be used for data analysis and visualization. It provides several in-built functions for data summarization and visualization such as pivot tables and charts.

4.R: This is a popular programming language used for statistical analysis and data visualization. It provides a wide range of packages for data exploration and visualization such as ggplot2, dplyr and tidyr.

Exploratory Data Analysis Techniques

Data Collection and Preparation: This is usually the first step in EDA. It involves identifying the data sources, gathering data and cleaning and transforming the data. Data cleaning involves removing or correcting any inconsistent, erroneous or missing data, while data transformation involves converting the data into a suitable format for analysis.
Data Summarization:This enables a data scientist to gain a better understanding of the data's main characteristics. It involves calculating summary statistics such as mean, median, mode, variance and standard deviation for each variable. In addition, frequency tables and histograms can be used to visualize the distribution of the data.

3.Data Visualization: This allows a data analyst to visually explore the data and identify patterns and trends. Various graphical techniques can be used to visualize the relationships between variables and detect any anomalies or outliers.

4.Data Exploration: This involves conducting further analysis on the data to identify patterns and relationships. This can be achieved by performing correlation analysis, regression analysis and factor analysis to identify the relationship between variables. In addition, clustering analysis can be used to group similar data points together and identify any underlying patterns.

Data Interpretation: Data interpretation involves making sense of the results obtained from data exploration and visualization. It involves interpreting the statistical significance of the results and identifying any meaningful relationships between variables.
The Process of Exploratory Data Analysis
Data Collection and Preparation:This is usually the first step. It may involve cleaning and transforming the data, handling missing values and outliers and selecting relevant variables for analysis.
Univariate Analysis: This involves examining individual variables in the data. This can be done by calculating summary statistics such as mean, median and standard deviation, then visualizing the distribution of the data using histograms, box plots and density plots. It is divided into two; non-graphical and graphical.

3.Bivariate Analysis: Involves examining the relationship between the variables in the data. This can be done by creating scatterplots, correlation matrices and heatmaps.

4.Multivariate Analysis: Involves examining the relationship between multiple variables in the data. This can be done by creating scatterplot matrices, principal component analysis and cluster analysis.

5.Data Visualization: This allows for the exploration of complex relationships and patterns in the data. This can be done using various graphical tools such as scatterplots, box plots, histograms and heatmaps.
EXAMPLE
In this example, we will be analyzing a dataset containing information about diamonds, including their carat weight, cut, color, clarity, and price.
This code loads the diamonds dataset, performs various data cleaning and exploration tasks, and creates several visualizations using MatplotLib and Seaborn. These visualizations include a histogram of carat weight, a boxplot of price by cut, a scatterplot of carat weight and price colored by cut, a correlation matrix of numerical variables, and a principal component analysis.

Import libraries

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

Load the dataset

diamonds = pd.read_csv('diamonds.csv')

View the first five rows of the dataset

print(diamonds.head())

Check for missing values

print(diamonds.isnull().sum())

Check for duplicates

print(diamonds.duplicated().sum())

Summary statistics of the dataset

print(diamonds.describe())

Histogram of the carat weight variable

plt.hist(diamonds['carat'], bins=30)
plt.xlabel('Carat Weight')
plt.ylabel('Frequency')
plt.title('Distribution of Carat Weight')
plt.show()

Boxplot of the price variable by cut

sns.boxplot(x='cut', y='price', data=diamonds)
plt.xlabel('Cut')
plt.ylabel('Price')
plt.title('Price by Cut')
plt.show()

_ Scatterplot of carat weight and price, colored by cut_

sns.scatterplot(x='carat', y='price', hue='cut', data=diamonds)
plt.xlabel('Carat Weight')
plt.ylabel('Price')
plt.title('Price vs. Carat Weight')
plt.show()

Correlation matrix of numerical variables

corr = diamonds[['carat', 'depth', 'table', 'price']].corr()
sns.heatmap(corr, cmap='coolwarm', annot=True)
plt.title('Correlation Matrix')
plt.show()

_ Principal component analysis_

from sklearn.decomposition import PCA
pca = PCA(n_components=2)
X = diamonds[['carat', 'depth', 'table', 'price']]
pca.fit(X)
X_pca = pca.transform(X)
plt.scatter(X_pca[:, 0], X_pca[:, 1], c=diamonds['cut'])
plt.xlabel('PCA 1')
plt.ylabel('PCA 2')
plt.title('PCA of Diamonds Dataset')
plt.show()

Python 101: Introduction to Python for Data Science

J Mungai — Sat, 18 Feb 2023 00:05:02 +0000

Python is a popular high level programming language used in various fields, including data analysis.
It has a simple and easy to learn syntax, making it a popular language among beginners and experienced alike.
Python is supported by a large community and has numerous open source libraries that make it ideal for data analysis.
Python libraries are collections of pre-written code that make certain tasks easier to perform.
Data analysis is essential in many industries including finance, marketing, healthcare and many more.
In this article, we will explore some of the essential concepts of Python for data analysis, including Python libraries, how to install Python for Windows, Python data types, Python data structures, Python functions, and Python programming basics.
**

LIBRARIES

**
Python's popularity in data science is due to its powerful and popular libraries. These libraries provide a set of tools and functions that make it easier to work with data and perform analysis tasks.
Some of the popular libraries for data analysis are:

1. NumPy
This is a python library used for numerical computations. It provides tools for creating and manipulating large and multi-dimensional arrays and matrices, hence making it useful for scientific computing and data analysis. NumPy has a comprehensive set of mathematical functions for linear algebra, Fourier transforms and random number generation.

2. Pandas
Pandas is a library for data manipulation and analysis. It does this by providing a set of functions for reading, group filter, reshaping, writing and processing tabular data. This makes Pandas useful for exploratory data analysis as well as easy to work with large datasets.

3. MatplotLib
This is Python's visualization library. It provides tools for creating different types of charts and plots such as bar charts, live graphs, histograms, heat maps and scatter plots. This makes for static, animated and interactive visualization of data.

4. Seaborn
This is a visualization library based on MatplotLib. It provides tools for creating complex visualization, such as heat maps, scatter plots, box plots, violin plots and pair plots. It is useful for creating publication quality visualizations.

5. SciPy
SciPy is built on NumPy. It is a library for scientific computing and technical computing. It is essential for scientific and engineering applications as it provides a set of tools and functions for optimization, integration, linear algebra and signal processing.

6.Scikit Learn
This is a Python library used for machine learning. It provides tools for building and evaluating machine learning models in Python.
Scikit has algorithms for classification, regression and clustering, which makes it a simple and efficient tool for data mining and data analysis.
**

How To Install Python For Windows

**
To install Python on Windows, you can follow these steps:

Go to the official Python website: https://www.python.org/downloads/
Click on the "Download Python" button. This will take you to a page with the latest versions of Python.
Choose the appropriate version of Python for your Windows operating system. If you're not sure which version to download, you can go with the latest version, which should work for most Windows systems.
Once you've downloaded the installation file, double-click on it to start the installation process.
Follow the installation wizard prompts. You can usually accept the default options for most of the prompts.
When the installation is complete, you can test that Python is installed correctly by opening a Command Prompt window and typing "python" (without the quotes) at the command prompt. If Python is installed correctly, you should see the Python version number displayed. That's it! You've successfully installed Python on your Windows computer. **

Python Data Types and Structures

**
Python has several built in data types:

1. Numbers
In Python, numbers are represented by three types: integers, floating-point numbers, and complex numbers. Integers are whole numbers with no decimal points, while floating-point numbers have decimal points. Complex numbers are made up of a real part and an imaginary part, and they are denoted using the "j" suffix.
For Example: Integers

my_integer = 42 print(my_integer) 42
Example: Float

price = 9.99 print(price) 9.99

2. Strings
Strings are sequences of characters enclosed in single or double quotes. They are used to represent text data. Python allows various operations on strings, such as concatenation, slicing, and formatting.
For example:
message = "Hello, world!" print(message) "Hello, world!"

3. Lists
Lists are ordered collections of elements, which can be of different data types. They are denoted using square brackets and can be modified after creation. Python allows various operations on lists, such as appending, removing, and slicing.
For example:

numbers = [1, 2, 3, 4, 5] print(numbers) [1, 2, 3, 4, 5]

4. Tuples
Tuples are similar to lists in that they are ordered collections of elements. However, unlike lists, tuples are immutable, which means that their elements cannot be modified once they are created. Tuples are denoted using parentheses.
For example:

fruits = ("apple", "banana", "orange") print(fruits) ('apple', 'banana', 'orange')

5. Sets
Sets are unordered collections of unique elements. They are denoted using curly braces and can be modified after creation. Python allows various operations on sets, such as union, intersection, and difference.
For example:

numbers = {1, 2, 3, 4, 5} print(numbers) {1, 2, 3, 4, 5}

6. Dictionaries
Dictionaries are collections of key-value pairs, where each key is associated with a value. They are denoted using curly braces and can be modified after creation. Python allows various operations on dictionaries, such as adding, deleting, and updating key-value pairs.
For example:
person = {"name": "John", "age": 35, "city": "Nairobi"} print(person) {'name': 'John', 'age': 35, 'city': 'Nairobi'}

7.Booleans
Boolean data types are a type of data that represents a logical value. In Python, the two Boolean values are 'True' and 'False'. Boolean values are often used in control structures, such as 'if' statements, to determine which branch of code should be executed.
Example of comparison of two values:

x = 10 y = 5 result = x > y print(result) # True

x = 5 y = 10 result = x > y print(result) # False