Exploratory Data Analysis (EDA)

In the early stage of a data analyst's career, it is preferable to start with baby step tasks and projects. Once you are conversant with the most basic of procedures employed in handling data, there you get confident in engaging in extra trivial analytics endeavors.
Beginning with a completed task brings joy and gratification, since it stirs achievement, not forgetting the fulfillment you get from doing something successfully.
A solider would tell you,"make your bed when you wake up. Go conquer the world, but if all goes wrong, at least you return to a nicely done bed ~ (Gratification)".

Exploratory Data Analysis s an approach to analyzing and summarizing data in order to understand its underlying patterns, relationships, and distributions. EDA is typically performed as a first step in the data analysis process, prior to any formal modeling or hypothesis testing.

EDA involves a wide range of techniques and methods for visualizing, summarizing, and exploring data, as the following.

1. Data cleaning and preprocessing: This involves identifying and handling missing or invalid data, removing duplicates, transforming variables, and more.

Data cleaning typically involves the following steps:

• Handling missing data: This involves identifying and handling missing data, such as imputing missing values, removing records with missing data, or using statistical methods to estimate missing values.

• Handling duplicates: This involves identifying and removing any duplicate records or observations in the dataset.

• Handling inconsistencies: This involves identifying and handling any inconsistencies in the data, such as misspellings, variations in formatting, or conflicting data.

• Handling outliers: This involves identifying and handling any outliers in the data, such as extreme values or data points that are significantly different from the rest of the data.

• Standardizing data: This involves converting data into a standard format, such as converting dates or times to a standard format or converting categorical data to numeric data.

• Handling errors: This involves identifying and correcting any errors in the data, such as data entry errors or data processing errors.

Data preprocessing typically involves the following steps:

• Data transformation: This involves converting data into a format that is more suitable for analysis or modeling, such as scaling numeric data, encoding categorical data, or reducing the dimensions in the data.

• Data normalization: This involves re-scaling data to a common scale or range, such as scaling numeric data to a range of 0 to 1 or standardizing data to have a mean of 0 and a standard deviation of 1.

• Data integration: This involves combining data from multiple sources or datasets into a single dataset.

• Data reduction: This involves reducing the size or complexity of the dataset, such as by using feature selection or feature extraction techniques to identify the most important features or variables.

• Data discretization (Discrete categories): This involves dividing continuous data into discrete categories or intervals, such as grouping education level data into categories of "early childhood", "primary", "junior high school", "senior high school" and "tertiary".

1. Descriptive statistics: This includes computing various summary statistics, such as mean, median, mode, variance, standard deviation, and more. This takes us back to your usual statistics lecture in college, a myriad of statistical terminologies. Let us walk down memory lane, hope have a refresher.

• Measures of central tendency: These are statistics that describe the typical or central value of a dataset. The three main measures of central tendency are the mean, median, and mode.

• Measures of variability: These are statistics that describe the spread or dispersion of a dataset. The most commonly used measures of variability are the range, variance, and standard deviation.

• Measures of shape: These are statistics that describe the shape of a distribution. Common measures of shape include skewness and kurtosis.

• Percentiles: These are statistics that divide a dataset into equal portions. For example, the median is the 50th percentile, meaning that 50% of the data falls below the median.

• Frequency distributions: These are tables or charts that display the frequency or count of each value or range of values in a dataset.

• Correlation coefficients: These are statistics that measure the strength and direction of the relationship between two variables. The most commonly used correlation coefficient is Pearson's correlation coefficient.

• Confidence intervals: These are statistics that provide a range of values within which a population parameter is likely to fall. Confidence intervals are often used to estimate the population mean or proportion based on a sample.

1. Visualization: This involves creating various charts and plots, such as histograms, box plots, scatter plots, heat maps, and more, to visualize the distribution, patterns, and relationships in the data. Unto them who have pictorial minds, here goes your candy jar. Diagrams spark remembrance together with the invaluable tool they are when one is making a presentation; to especially novices in the business fields.

Data visualization is important in data analysis and communication because it can help to uncover patterns, trends, and relationships that might not be immediately obvious from looking at raw data. By presenting data in a visual format, data visualization can also make it easier for people to understand and interpret complex information, and to identify important insights and opportunities.

There are many different types of data visualization techniques that can be used depending on the type of data and the intended audience. Some common types of data visualizations include:

• Bar charts and histograms: These are used to display the distribution of data across different categories or ranges.
• Line charts: These are used to show trends or changes in data over time.
• Scatterplots: These are used to show the relationship between two variables.
• Heat maps: These are used to show the distribution of data across two or more dimensions using color coding.
• Pie charts: These are used to show the proportion of data within different categories.
• Box plots: These are used to show the distribution of data along with any outliers or extremes.

1. Dimensionality reduction: This involves reducing the number of variables or features in the data, through techniques such as principal component analysis (PCA), factor analysis, and more.

2. Clustering and classification: This involves grouping or categorizing data into meaningful clusters or categories based on their similarities or differences, using techniques such as k-means clustering, hierarchical clustering, and more.

3. Correlation and regression analysis: This involves identifying and measuring the relationships between variables, using techniques such as correlation analysis, linear regression, logistic regression, and more.

Overall, EDA is a crucial step in the data analysis process, as it allows data scientists and analysts to gain a deeper understanding of the data, identify potential issues or biases, and generate hypotheses for further testing