DEV Community: Joseous Ng'ash

The Importance of Hypothesis Testing in Data Science.

Joseous Ng'ash — Thu, 02 Jul 2026 02:27:28 +0000

Data Science: To a beginner most of the time it means data collecting and building machine models but it is more than that. It is about making informed decisions based on evidence.
Whether you are analyzing customer behavior, testing medical treatment, evaluating marketing campaign or improving business processes, Hypothesis testing provides a scientific framework for whether observed patterns are genuine or simply due to random chance.
To avoid drawing unreliable conclusions, Hypothesis testing allows scientists to validate assumptions, compare groups, measure relationship and support decision-making with statistical confidence.
In this article, we will explore the importance of hypothesis testing and examine the relevance of every major hypothesis test commonly used in Data Science.

What is Hypothesis Testing?

Hypothesis testing is a statistical method used to determine whether there is enough evidence in a sample to support a claim about an entire population.
Every hypothesis test begins with two statements:

Null Hypothesis(H0): The default assumption that there's no effect, no difference or no relationship in the population. You assume this is true until the data proves otherwise
Alternative Hypothesis(H1): Your research assumption: It states that there's an effect, a difference or a relationship. This is what you are actually trying to prove

Example:

H0: A new recommendation algorithm does not increase customer purchases.
H1: A new recommendation algorithm increases customer purchases. The decision is based on a P-Value, which measures the probability of observing the sample results if the null hypothesis were true.
Typically: if p-vale <= alpha (0.05), reject H0

Why Hypothesis Testing Matters in Data Science

Hypothesis testing is important because it helps data scientist:

Detect meaningful relationships
Evaluate business changes
Validate machine learning results
Compare algorithms
Build confidence in analytical findings
Make objective decisions

Instead of saying 'Sales appear to have increased', hypothesis test allows us to conclude, 'Sales increased significantly with a p-value of 0.01'.
This distinction is what separates descriptive analysis from evidence-based decision making.

Steps in Hypothesis testing/Best practices

Step 1: Define the business question
Step 2: States the hypothesis - Clearly define both Null and alternative hypothesis.
Step 3: Formulate - Decide the level of significance(alpha) and the statistical test(eg t-test, chi-square,ANOVA)
Step 4: Analyze the data - Calculate the test statistic and its corresponding p-value from the sample data.
Step 5: Interpret the results - Compare the p-value to your significance level.
- If p-value <= alpha, reject the null hypothesis (p <= 0.05)
- If p-value > alpha, fail to reject the null hypothesis

Types of Hypothesis Tests and Their Importance

1. T-Test

we have various types of t-tests:

One-Sample ttest
Paired ttest
Independent ttest

1.1 One-Sample T-test

Compares a sample mean to a known population mean.

Example

The average height of Kenyan men is 5'5 ft.

H0 : The average height of kenyan men is 5'5 ft(u = 5'5 ft)
H1: The average height of kenyan men is not equal to 5'5 ft( u != 5'5 ft)

Assumptions of One sample ttest

Continuous scale: the data must be continuous
Random sampling: Data must be collected from random sample of the population
Normality: The data in the population should be approximately normally distributed

Real world Example:

from scipy.stats import ttest_1samp

# Two tailed test: Is the average rent in Nairobi equal to 120000

h0 = "Average rent is equal 120000(u = 120000)"
h1 = "Average rent is not equal to 120000(u != 120000)" 

alpha = 0.05

stat, p = ttest_1samp(housing_data["monthly_rent_kes"], popmean=120000)

print(f"T-statistics: {stat:.2f}")
print(f"P-Value: {p:.2f}")

if p <= alpha:
    print(f"Reject the null hypothesis {h0}")
else:
    print(f"Fail to reject the null hypothesis {h0}")

#Ouput
T-statistics: 25.53
P-Value: 0.00
Reject the null hypothesis Average rent is equal 120000(u = 120000)

Importance in Data Science

Business KPIs
Quality control
Website performance metrics
Manufacturing benchmarks

1.2 Independent T-test

compares the means of two independent groups

Assumptions

Continous variable: The dependent variable must be measured on a continous scale
Two groups: The independent variable must consist of exactly two categorical, independent groups
Independent observations: There must be no relationship between the participants in either group
Normal distribution: The depedent variable must be approximately/roughly normally distributed within each of the two groups
Random Sampling: Data must be collected using a random sampling method from the population.
Homogeneity of variance: The variance(spread) of the independent variable must be roughly in both groups

Example

A bank wants to know whether people who are approved for loans have a significant different average salary compared to people who are not approved.

Group 1: Customers whose loans have been approved
Group 2: Customers whose loans have not been approved We are comparing their salaries

from scipy.stats import ttest_ind

h0 = "There is no significant difference between the salaries in Group 1 and 2(u1 = u2)"
h1 = "The Average salaries for Group1 and Group2 are not equal"

data = {"customer":["Mark", "Steven", "Allan", "Bill", "Micheal", "Jesse", "Sam"],
        "salary":[85000, 92000, 78000, 110000, 97000, 45000, 52000],
        "loan approval":["Approved", "Approved", "Not Approved", "Not Approved", "Not Approved", "Approved", "Approved"]}

df = pd.DataFrame(data)

approved_salary = df[df["loan approval"] == "Approved"]["salary"]
not_approved_salary = df[df["loan approval"] == "Not Approved"]["salary"]

alpha = 0.05

t_test, p = ttest_ind(approved_salary, not_approved_salary, equal_var=False)

print("T-statistic", t_test)
print("P-value", p)

if p <= alpha:
    print(f"Reject null hypothesis: {h0}")
else:
    print(f"Fail to reject null hypothesis: {h0}")

#Output
T-statistic -1.7715946167319203
P-value 0.13671546072156335
Fail to reject null hypothesis: There is no significant difference between the salaries in Group 1 and 2(u1 = u2)

Importance in data science

Independent T-test is one of most common tests in business analytics and it is used for:

Medical research
Marketing experiments
Customer segmentation
Product comparison

1.3 Paired T-test

Compares the same users before and after a product change or a system change.

Assumptions

Continous dependent variable: The dependent variable must be measured on continous scale
Paired observations: The data must consist of matched pairs
Normal distribution of differences: The differences of the pairs must follow a normal distribution

Note: The individual group do not need to have a normal distribution, only the calculated difference

Example

Netflix upgrades its recommendation algorithm.

The same users are observed before and after the upgrade

user	Before(minutes watched)	After(minutes watched)
1.	65	81
2.	44	59
3.	73	75

Research question
Did the new recommendation system increase with time?

from scipy.stats import ttest_rel

data = {"User":[1, 2, 3],
        "Before":[65, 44, 73],
        "After":[81, 59, 75]}

netflix_df = pd.DataFrame(data)

# formulate the hypothesis

H0 = "There's no difference in watch time before and after new recommendation system(u1 = u2)"
H1 = "There's a significant difference in the watchtime before and after"

alpha = 0.05

stat, p = ttest_rel(netflix_df["Before"], netflix_df["After"])

print("T-statistic", stat)
print("P-value", p)

if p <= alpha:
    print(f"Reject null hypothesis: {H0}")
else:
    print(f"Fail to reject null hypothesis: {H0}")

# Output
T-statistic -2.4394301941500904
P-value 0.13486786659642408
Fail to reject null hypothesis: There's no difference in watch time before and after new recommendation system(u1 = u2)

Paired T-test is important because is perfect for measuring improvements.

2. Z-Test

Purpose:
Z-test tests population means when:

Sample size is large
Population variance is known

Assumptions required to use Z-test

Known population variance or standard deviation: The true population variance and standard deviation must be known
Random sampling: Your data must be collected using a simple random sampling method to ensure it is representative of the broader population
Independence of observation: The data point within the sample or between the samples must be entirely independent of one another
Normality: The underlying population should follow a normal distribution
Continuous data: The variable being measured should be continuous rather than discrete

3. Chi-Square Test

The Chi-Square test is used when working with categorical data.
It helps answer questions such as:

Is a disease related to smoking status?
Is customer gender related to the product preference?

There are two common Chi-Square test

Chi-Square test of independence
Chi-Square goodness of fit test

3.1 Chi-Square test of independence

Used to determine whether two categorical variables are related. eg Is a customer's gender associated with whether they purchase a product

Gender	Product Preference
Male	Electronic appliances
Female	Fashion
Female	Beauty
Male	Camping

Null Hypothesis(H0): There is no relationship between customer's gender and their product preference
Alternative Hypothesis(H1): There is a relationship between customer's gender and their product preference.

example

Imagine you are a data scientist and you'd like to investigate:

Is a student's county associated with whether they secure employment after completing a data course?

from scipy.stats import chi2_contingency

data = {"county":["Nairobi", "Nairobi", "Nairobi", "Nairobi", "Kiambu", "Kiambu",
                  "Kiambu", "Kisumu", "Kisumu", "Kisumu", "Nakuru", "Nakuru"],
                  "employed":["Yes", "Yes", "No", "Yes", "Yes", "No", "No", "Yes", "No", "No",
                              "Yes", "Yes"]}
H0 = "There is no relationship between student's county and employment"
H1 = "There is a relationship between student's county and employment"

alpha = 0.05

table = pd.crosstab(df["employed"], df["county"])

chi2, p, dof, expected = chi2_contingency(table)

print(display(table))

print("Chi-Square:", chi2)
print("P-Value:", p)
print("Degree of freedom:", dof)
print("Expected Value:", expected)

if p <= alpha:
    print(f"Reject Null hypothesis: {H0}")
else:
    print(f"Fail to reject the null hypothesis: {H0}")
df = pd.DataFrame(data)

# Output:
None
Chi-Square: 3.4285714285714284
P-Value: 0.3301450108968586
Degree of freedom: 3
Expected Value: [[1.25       1.25       1.66666667 0.83333333]
 [1.75       1.75       2.33333333 1.16666667]]
Fail to reject the null hypothesis: There is no relationship between student's county and employment

3.2 Chi-Square goodness of fit

Used to observe or determine whether the observed frequencies match the expected frequencies

Null Hypothesis(H0): Observed frequencies match the expected frequencies
Alternative Hypothesis(H1): Observed frequencies differ from expected frequencies eg Do customers choose products in the proportions we expected?
Null Hypothesis(H0): Customers choose products in the expected proportions
Alternative Hypothesis(H1): Customers did not choose products in the expected proportions

Example

A supermarket expects sales to be evenly distributed among four products.

Expected sales:

Product	Expected sales
Colgate	25
Pepsodent	25
Aquafresh	25
Sensodyne	25

Actual sales:

Product	Observed sales
Colgate	35
Pepsodent	20
Aquafresh	30
Sensodyne	15

Is the difference significant?

from scipy.stats import chisquare 

sales_data = {"product":["Colgate", "Pepsodent", "Aquafresh", "Sensodyne"],
             "expected sales":[25, 25, 25, 25],
             "observed sales":[35, 20, 30, 15]}

sales_df = pd.DataFrame(sales_data)

# Formulate the hypothesis

H0 = "Observed frequencies match the expected frequencies.(O = E)"
H1 = "Observed frequencies differ from expected frequencies"

alpha = 0.05

observed = sales_df["observed sales"]
expected = sales_df["expected sales"]

chi2, p = chisquare(f_obs=observed, f_exp=expected)

print("Chi-square: ", chi2)
print("P-value: ", p)

if p <= alpha:
    print(f"Reject the null hypothesis: {H0}")
else:
    print(f"Fail to reject null hypothesis: {H0}")

#Output:
Chi-square:  10.0
P-value:  0.01856613546304325
Reject the null hypothesis: Observed frequencies match the expected frequencies.(O = E)

4. ANOVA(Analysis Of Variance)

ANOVA is a statistical method used to determine if there are significance differences between the means of three or more independent groups.(Categorical vs numerical).
It simply compares means across three or more groups.

Two common types of ANOVA

One-Way ANOVA.
Two-Way ANOVA.

Note: In this article we will focus on One-Way ANOVA because it is most used ANOVA test.

Assumptions of ANOVA
For your ANOVA test to be valid, your data must meet three core asumptions.

Independence: The observations in each group are independent of one another and the participants are randomly sampled.
Normality: The dependent variable is approximately normally distributed within each group.
Homogeneity of variances: The variance in each group should be roughly equal.

ANOVA is Widely used for:

Manufacturing
Education research
Product testing
Medical trials

4.1 One-Way ANOVA

It compares one independent variable across multiple groups.

Practical Example of One-Way ANOVA Test

# Housing Example: Do average rents differ across estates

from scipy.stats import f_oneway 

#Formulate the hypothesis
h0 = "There is no difference in rents across estates"
h1 = "Atleast one average rent is different"

alpha = 0.05

groups = [group["monthly_rent_kes"].values for _, group in housing_data.groupby("estate")]

f_stat, p_val = f_oneway(*groups)
print("F-Statistic", f_stat)
print("P-Value", p_val)

if p <= alpha:
    print(f"Reject null hypothesis: {h0}")
else:
    print(f"Fail to reject null hypothesis: {h0}")

It helps determine if there is any significant difference in rent across estates.

5. Mann_Whitney U Test

It is Non-parametric alternative to the independent t-test.
Instead of comparing the mean of two independent groups, it compares the rankings of observations between those groups.
The test is useful when data is not normally distributed

Difference between Mann_Whitney U Test and Independent T-test

Mann-Whitney U test => "Is the observation from one group generally higher or lower than those from the other group?"
Independent t-test => "Is the everage value between the two groups different?"

When to use Mann-Whitney U test

There are two independent groups.
The response variable is ordinal or not normally distributed.
When the assumptions of independent ttest are violated.

Practical Example of Mann-Whitney U Test

from scipy.stats import mannwhitneyu
# housing exmple: Is there a significant difference in monthly rent between properties in Karen and Umoja

h0 = "There is no significant difference in monthly rent in Umoja and Karen"
# Monthly distribution is the same in karen and Umoja
h1 = "There is a difference in monthly rent in Umoja and Karen"
# monthly distribution differs between Karen and Umoja

alpha = 0.05

karen_rent = housing_data[housing_data["estate"] == "Karen"]["monthly_rent_kes"]
umoja_rent = housing_data[housing_data["estate"] == "Umoja"]["monthly_rent_kes"]

stat, p_value = mannwhitneyu(karen_rent, umoja_rent, alternative="two-sided")

print("Karen mdeian rent:", karen_rent.median())
print("Umoka mdeian rent:", umoja_rent.median())

print("U-Statistic: ", stat)
print("P-Value ", p_value)

if p <= alpha:
    print(f"Reject null hypothesis '{h0}'")
else:
    print(f"Fail to reject null hypothesis '{h0}'")

# Output
Karen median rent: 363740.0
Umoja median rent: 96559.5
U-Statistic:  15232.0
P-Value  5.768145774470841e-42
Fail to reject null hypothesis 'There is no significant difference in monthly rent in Umoja and Karen'

6. Wilcoxon Signed-Rank Test

The Wilcoxon Signed-Rank Test is the non-parametric equivalent to the paired t-test.
It compares paired observations, such as measurements taken from the same subjects before and after intervention.
Rather than comparing the mean of the paired differences, it ranks the absolute differences(while keeping track of their signs) to determine whether there is a systematic increase or decrease.

Difference between Wilcoxon Signed-Rank Test and Paired T-test

The paired ttest => "Did the average value change"
The Wilcoxon signed-rank test asks => "Did the typical(median) paired observations changed?"

When is Wilcoxon Signed-Rank Test used?
Use this test when:

The same individuals are measured twice.
The difference between paired observations are not normally distributed
The response is ordinal or continuous

Practical Wilcoxon Signed- Rank Test

# Housing example: Did property rents change significantly from 2024 to 2025?
from scipy.stats import wilcoxon

H0 = "There is no significant change between rent in 2024 and rent in 2025"
H1 = "There is a significant difference between the rent in 2024 and rent in 2025"

alpha = 0.05

rent_2024 = housing_data["rent_2024"]
rent_2025 = housing_data["rent_2025"]

stat, p_value = wilcoxon(rent_2024, rent_2025, alternative="two-sided")

print("Test statistic: ", stat)
print("P-Value: ", p_value)

if p <= alpha:
    print(f"reject null hypothesis '{H0}'")
else:
    print(f"Fail to reject null hypothesis '{H0}'")

# Output
Test statistic:  0.0
P-Value:  0.0
Fail to reject null hypothesis 'There is no significant change between rent in 2024 and rent in 2025'

It is useful when the data is not normally distributed.

7. Kruskal-Wallis Test

The Kruskal-Wallis test is the equivalent of the one-way ANOVA Test.
It compares three or more independent groups by ranking all observation together and then determine whether one or more groups tend to have higher or lower ranks than others.

Difference between One-Way ANOVA and Kruskal-Wallis Test

One-Way ANOVA Test => "Do the average values differ among these groups?"
Kruskal-Wallis Test => "Do the distribution(or typical ranks) differ among these groups?"

When is Kruskal-Wallis Test used?

There are three or more independent groups.
The response variable is ordinal or not normally distributed.
When the assumptions of ANOVA are violated

Practical Example using Kruskal-Wallis Test

# Housing Example: Does the monthly rent differ across properties with different numbers of bedrooms?
from scipy.stats import kruskal

H0 = "Monthly distribution is the same across all bedroom groups"
H1 = "Atleast one bedroom group has a differfent rent distribution"

alpha = 0.05

groups = [group["monthly_rent_kes"].values for name, group in housing_data.groupby("bedrooms")]

stat, p_val = kruskal(*groups)

print("Kruskal wallis H statistic: ", stat)
print("P-Value: ", p_val)

if p <= alpha:
    print(f"Reject null hypothesis '{H0}'")
else:
    print(f"Fail to reject null hypothesis '{H0}'")

# Output
Kruskal wallis H statistic:  135.76123559380773
P-Value:  1.4235272194460855e-27
Fail to reject null hypothesis 'Monthly distribution is the same across all bedroom groups'

Choosing the Right Hypothesis Test

Goal	Recommended Test
Compare one mean to a known value	One-Sample t-Test
Compare two independent groups	Independent t-Test
Compare before and after measurements	Paired t-Test
Compare three or more groups	ANOVA Test
Compare three or more non-normal groups	Kruskal-Wallis Test
Compare categorical variables	Chi-Square Test

Hypothesis Testing in Real-World Data Science

Hypothesis testing is applied across many industries:

E-Commerce
Manufacturing
Healthcare
Financial Industries
sports analysis
Marketing
Education sector
Human resources

Conclusion

Hypothesis testing being the cornerstone of data science help in transforming raw data into evidence that can support confident decisions.
Each hypothesis tests method serves a distinct purpose in the analytical workflow. Understanding when to use each test, why its appropriate and how to interpret its results enables data scientist build more trustworthy analyses, improve predictive models and deliver insights that drive meaningful business outcomes.
Successful data scientists use hypothesis tests as complementary tools for exploring data, validating assumptions and making evidence-based decisions that stand up to scrutiny.

Understanding Data Distributions and Their Impact on Data Science.

Joseous Ng'ash — Wed, 24 Jun 2026 10:08:45 +0000

Data is at the heart of every data science. Whether you are predicting market growth, analyzing customer behavior or building machine models, the way data is distributed significantly affects the insights and outcomes obtained.

Most if not all beginners focus heavily on cleaning data and building models but mostly overlook one of the most crucial concepts in statistics and data science: Data Distributions.
Data scientist needs to understand data distributions to help identify patterns, detect anomalies, select appropriate statistical techniques and improve machine learning performance.

In this article, we will explore data distributions, common types of distributions, how to visualize them and why they matter in data science.

What is Data Distribution?

A data distribution is a function or a set of graphical representations that shows all the possible values a variable can take and how frequently those values occur.

A distribution shows us what the data forms. It shows:

The overall shape of the data
The presence of outliers
Whether the data is symmetrical or skewed
How values are dispersed
Where most observations occur

For Example, consider exam scores from 100 students:

Score Range	Number of Students
0-20	5
21-40	15
41-60	35
61-80	30
81-100	15

Most students scored between 41 to 80, indicating where the data is concentrated

Why Distributions Matter in Data Science

Since many statistical and machine learning method assume specific data characteristics, understanding distributions is paramount

Distributions helps in:

1. Detect Outliers
Outliers can significantly affect statistical calculations and machine learning algorithms

salary = [30000, 35000, 40000, 45000, 50000, 1000000]

The salary of 1,000,000 is an outlier that can distort averages and predictions.

2. Understand Data Behavior
You need to understand how your data behaves before building models.
Distributions answers:

Is the data balanced?
Are there extreme values?
Is the data centered around a mean?

3. Choose Appropriate Models
Some algorithms assume normal distributed data.

Example include:

T-tests
Logistic Regression
Linear Regression
ANOVA

To know if data transformation is necessary, we must understand distributions.

4. Improve Feature Engineering
Features with highly skewed distributions often benefit from transformations such as:

Square root transformation
Log transformation
Box-Cox transformation These transformations can improve model performance

Common types of Data Distributions

1. Normal Distribution

This is the most famous type of distribution in statistics. It is often called the Bell Curve because of its shape.

Characteristics:

Symmetrical - left side mirrors the right side
Mean = Median = Mode
Most values cluster near the center
Extreme values are rare

Examples:

IQ scores
Measurement errors
Human height

Normal distribution visual

Many statistical techniques assume normality because it simplifies analysis and prediction.

2. Uniform Distribution

Uniform distribution is Where every possible value has an exactly equal chance of occurring.

Example:
Rolling a fair side dice

Each number:

1, 2, 3, 4, 5, 6

Has the same chance of appearing

Uniform distribution visual

Uniform distribution commonly appear in simulations and random sampling.

3. Right-Skewed Distribution

Tail stretches to the right. In most occasions the mean is greater than the median. It is also called a Positive skewed distribution.
The majority of values are small and only few very large values exist.

Example:

Website traffic
Income
House prices

Right-Skewed distribution

Impact on Data Science

Right-skewed data can:

Affect model assumptions
Require transformation
Inflate means A log transformation often helps normalize such data

import numpy as np

df["income_log"] = np.log(df["income"])

4. Left-Skewed Distribution

Tail stretches to the left. It is also known as Negative skewed distribution.
The median is more robust statistic in the presence of extreme values.
In this distribution, most values are high and few unusually small values exist.

Example:

Product ratings for highly rated products
Easy exam scores

Left-Skewed distribution visual

Left skewed distribution may require transformation or special treatment when statistical assumptions are violated.

3. Bimodal Distribution

A bimodal distribution is a statistical dataset or probability distribution with two distinct peaks (modes) where values occur most frequently.
Bimodal distributions often indicate that multiple populations exist within the same dataset which can suggest the need for segmentation or clustering.

Example:

Customers age groups
Product usage patterns
Heights of men and women combined

Bimodal distribution visual

Visualizing Distributions

We use visualization as one of the easiest way to understand data distribution

Histogram

Histograms display frequencies of data values.

# Rent distribution
# Confirm if the mean is greater than the median

median_rent = housing_data["monthly_rent_kes"].median()
mean_rent = housing_data["monthly_rent_kes"].mean()

print(f"Median Rent {median_rent} vs Mean Rent {mean_rent}")

# Plot the distribution

plt.figure(figsize = (6,2))

sns.histplot(housing_data, x="monthly_rent_kes")
plt.title("Monthly rent distribution")
plt.show()

Output:

Advantages:

Reveals skewness
Shows overall shape
Easy to interpret

Box Plot

Box plots summarize distribution using:

Median
Quartiles
Outliers

Box Plot representation

Example of plotted Box plot

# satisfaction score by furnishing status

plt.figure(figsize=(6,3))

sns.boxplot(data=housing_df, x="furnishing", y="satisfaction_score")
plt.ylabel("Satisfaction Score")
plt.xlabel("Furnishing Status")
plt.show()

Output

Key Distribution Metrics

a). Mean
The average value

ages = [23, 24, 25, 26, 34, 21, 30]
avg_age = np.mean(ages)
print(avg_age)

Mean is sensitive to outliers.

b). Median
This is the middle value.

ages = [23, 24, 25, 26, 34, 21, 30]

median_age = np.median(ages)
print(median_age)

Median is more robust than the mean.

c). Variance
Variance measures how spread out data is.

data = {"Age": [25, 30, 35, 40],
        "Salary": [50000, 54000, 62000, 68000]
        }

df = pd.DataFrame(data)

age_var = df["Age"].var()

print(f"Age Variance: {age_var}")

Higher variance means greater dispersion.

d). Standard Deviation
Standard deviation represents the average distance from the mean.

floor_size_std = housing_data["floor_size_sqm"].std()

print(f"The standard deviation: {floor_size_std}")

Standard deviation is widely used in statistical analysis and machine learning.

e). Skewness
This metric measures asymmetry of a probability distribution around its mean.

skewness = df["salary"].skew()

Interpretation:

0 --> Symmetrical
Positive --> Right-skewed
Negative --> Left-skewed

Impact of Distributions on Machine Learning

Model performance is directly influenced by distributions

Feature Scaling
Highly skewed features can dominate learning algorithms.
Common scaling techniques:

from sklearn.preprocessing import StandardScaler

scaler = StandardScaler()
scaled_data = scaler.fit_transform(df)

Data Transformation
when distributions are heavily skewed, they reduce skewness, improves model assumptions and stabilizes variance

import numpy as np

df["price_log"] = np.log1p(df["price"])

Outliers Detection
Distributions help identify unusual observations.

Methods include:

Interquartile range (IQR)
Isolation Forest
Z-score

Example:

# Example monthly rent kes
# First quartile
q1 = housing_data["monthly_rent_kes"].quantile(0.25)

# third quartile
q3 = housing_data["monthly_rent_kes"].quantile(0.75)

# Interquartile range 
iqr = q3 - q1

print(f"Q1 rent: {q1}")
print(f"Q3 rent: {q3}")
print(f"IQR : {iqr}")

#Output
Q1 rent: 129188.25
Q3 rent: 287698.25
IQR : 158510.0

This helps identify values outside the normal range.

Best Practices When Working with Distributions

Before building any model:

Visualize your data
Check for skewness
Identify outliers
Understand feature spread
Apply transformations when necessary
Verify assumptions of statistical methods
Compare mean and median

These simple steps can prevent many common data science mistakes.

Conclusion

Data science is built on foundational concepts of Data Distributions. Data distributions provides valuable insights into how data behaves, helping analysts and data scientists make informed decisions throughout the analytical process.

By understanding distributions, you can:

Create more accurate predictive models
Select appropriate statistical methods
Better explore datasets
Improve machine learning performance
Detect anomalies and outliers

When undertaking any data science project, the best approach is to take your time to understand data distribution, it may reveal insights that significantly influence your results.
By understanding your data, you can be able to build better algorithms.

Pandas and Data Visualization Using Matplotlib and Seaborn

Joseous Ng'ash — Sat, 20 Jun 2026 19:08:07 +0000

New chapter in Learning data analytics and data science. The focus now is on Pandas as a Python library alongside Matplotlib and Seaborn for data visualization.
Am writing this article to guide beginners who are already or beginning the data analytics and data science profession.

Introduction to Python Data Analysis

In modern world Data has become most valuable asset. Business, healthcare institutions, financial and even social media platforms rely heavily on data to make informed decisions.
Raw data is of no use and that where Data Analysis and visualization becomes important.

One of most used programming language is Python for data analysis because of its simplicity and powerful libraries.
Commonly used libraries are Pandas, Matplotlib and Seaborn.
Pandas: Helps in cleaning, organizing and analyzing data.
Matplotlib and Seaborn: are used to create visual representations of data.

This article introduces Pandas and explains how Matplotlib and Seaborn can be used for effective data visualization in a begginer friendly way.

What is Pandas and Why it Matters

Pandas is open source Python Library used for data manipulation and analysis. It provides simple and efficient tools for working with structured data eg CSV, spreadsheets and databases.

Main data structures in Pandas are:

Series: it is a one-dimensional array like structure.

import pandas as pd
age = {"Age":[18, 20, 23, 40, 50, 24]} 

series = pd.DataFrame(age)
series

#Output: 
Age
0   18
1   20
2   23
3   40
4   50
5   24

DataFrame: It is two dimensional table similar to an Excel spreadsheet or SQL table.

students = {"Name":["Mark", "John", "Nancy"],
            "Grade":["A", "B", "C"],
            "Course":["Data Science", "Data Engineering", "Data Analytics"]}

student_grades = pd.DataFrame(students)
student_grades

#Output: 
    Name    Grade   Course
0   Mark    A   Data Science
1   John    B   Data Engineering
2   Nancy   C   Data Analytics

Pandas matters because it simplifies complex tasks such as:

Performing calculations and statistical analysis
Filtering and sorting information
Cleaning missing or incorrect data
Reading datasets from files

Before using Pandas, it must be installed, run the following command:

pip install pandas

Then imported into Python script:

 import pandas as pd

Reading Data Pandas can easily read files such as CSV and Excel Example:

import pandas as pd

df = pd.read_csv("students.csv")
print(df.head())

The head() function displays the first five rows of the dataset.

Checking Data Information When you want to understand the structure of your data:

print(data.info()) 
print(data.describe())

info() shows column names, data types and missing values
describe() provides statistical summaries such as maximum values, averages

Handling Missing Values Missing values can affect the analysis results. Checking missing data

 print(data.isnull().sum())

Removing missing values:

data = data.dropna()

Filling missing values:

data["Age"] = data["Age"].fillna(data["Age"].mean())

This replaces missing age values with the average age.
Filtering allows users to select specific information
Example:

high_scores = data[data["Score"] > 70] 
print(high_scores)

Sorting data:

sorted_data = data.sort_values(by="Score", ascending=False) 

print(sorted_data)

These operations help organize data for better understanding and reporting

Data Visualization Fundamentals

Data Visualization: It is the process of representing data graphically using charts, graphs and plots. Visualization makes it easier to identify patterns, trends and relationship in data.

For Example:

Scatter plots shows relationships between variables
Bar Charts compares category
Pie Chart display proportions
Line Charts show trends over time

Visualizations helps understand large datasets because humans interpret visuals faster than raw numbers.

Python provides powerful visualization libraries, with Matplotlib and Seaborn being among the most widely used.

Using Matplotlib for Charts

Matplotlib is one of the oldest and most flexible visualization libraries in Python. It provides full control over the chart customization

To install Matplotlib

pip install matplotlib

Import it:

import matplotlib.pyplot as plt

Creating a Line Chart
A line chart is used to show trends.
Example:

import pandas as pd
import matplotlib.pyplot as plt 
plt.figure(figsize=(6, 3))

sns.lineplot(data=housing_df, x="bathrooms", y="bedrooms")
plt.title("Bathrooms vs Bedrooms")
plt.xlabel("Bathrooms")
plt.ylabel("Bedrooms")
plt.show()

The chart shows relationship between Bathrooms and bedrooms.

Creating Bar Chart
Bar chart compare categories.
Example:

# Average satisfaction score by property type
avg_satisfaction_by_prop = housing_df.groupby("property_type")["satisfaction_score"].mean().sort_values(ascending = False).reset_index()

#Plot
plt.figure(figsize = (6,3))

sns.barplot(data = avg_satisfaction_by_prop, x = "property_type", y = "satisfaction_score")
plt.title("Average Satisfaction Score by Property type")
plt.xlabel("Property Type")
plt.ylabel("Average Satisfaction Score")
plt.show()

The chart compares Average satisfaction per property type

Creating Pie Chart
Pie charts represents percentages.
Example:

furnishing_counts = housing_df["furnishing"].value_counts()

explode = (0.05, 0.05, 0.05)

plt.figure(figsize = (6, 6))

plt.pie(furnishing_counts, explode = explode, labels = furnishing_counts.index, autopct="%1.1f%%")
plt.title("Distribution Of the furnishing status")
plt.show()

Pie Chart:

Matplotlib is highly customizable and allows users to change colors, labels, chart sizes, and grid styles.

Using Seaborn for Statistical Visualizations

Seaborn: It is Python library built on top of Matplotlib. It provides more attractive and advanced statistical visualizations with less code.

Install Seaborns

pip install seaborn

Import it:

import seaborn as sns

Seaborn works smoothly with Pandas DataFrames.

Example dataset:

import pandas as pd 
data = { "Student": ["John", "Mary", "Peter", "James"], "Score": [85, 90, 78, 88] }
df = pd.DataFrame(data)

Bar Plot

# Average monthly rent by property type
plt.figure(figsize=(6, 3))

sns.barplot(data=housing_df, x = "property_type", y = "monthly_rent_kes", estimator = "mean", palette = "bright")
plt.title("Average monthly rent by property type")
plt.xlabel("Property Type")
plt.ylabel("Average monthly rent")
plt.xticks(rotation=45)
plt.show()

Seaborn automatically applies better styling than in matplotlib.
Example of more styled Bar Plot:

Histogram
Histograms show data distribution.
Example:

# What is the distribution of monthly rent
plt.figure(figsize = (6, 3))

sns.histplot(data=housing_df, x = "monthly_rent_kes")
plt.title("Distribution of monthly rent")
plt.xlabel("Monthly rent")
plt.ylabel("Number of properties")
plt.show()

This helps determine distribution of Monthly Rent.

Scatter Plot
It reveals relationships between variables.
Example:

# Plotting relationship between bedrooms and bathrooms

plt.figure(figsize=(6, 3))

sns.scatterplot(data=housing_df, x="bathrooms", y="bedrooms")
plt.title("Bathrooms vs Bedrooms")
plt.xlabel("Bathrooms")
plt.ylabel("Bedrooms")
plt.show()

This chart helps to show the relationship between bathrooms and bedrooms.

Heatmap
Heatmaps show relationship between numerical values.
Example:

# Correlation analysis

correlation = housing_df[numerical_columns].corr()

plt.figure(figsize=(6,6))

sns.heatmap(correlation, annot=True, fmt=".2f")
plt.title("Correlation Heatmap for Numerical Variables")
plt.show()

They helps identify strong or weak relationships in datasets.

Heatmap Example

Matplotlib is suitable when detailed customization is required, while Seaborn is ideal for creating visually appealing statistical charts quickly.
In real world, most Analysts use both libraries together because seaborn is built on top of Matplotlib.

Best Practices and Common Mistakes.

Best Practices:

Always clean data before analysis
Use of appropriate charts
Add labels and titles to charts
Visualization should be simple and easy to read
Check for missing or duplicate data

Common Mistakes:

Forgetting labels and or legends
Ignoring missing values
Using wrong charts
Overcrowding charts with too much information

Clear visualization communicates datasets information effectively without confusion.

Conclusion

For analyst to created clear and understandable visualizations, they must use Pandas, Matplotlib, and Seaborn.

Pandas simplifies data cleaning, manipulation, and analysis.
Matplotlib and Seaborn transform raw numbers into meaningful visual insights.

For data analyst or data science beginners, learning these libraries is essential because they are widely used in industries like finance, healthcare, marketing and business intelligence.
Mastering these tools is an important step toward becoming a skilled analyst or data scientist.

Python and How Python Is Used In The Data Analytics Space. A Beginner's Guide.

Joseous Ng'ash — Fri, 15 May 2026 02:05:22 +0000

Introduction

In today's digital world, data is everywhere, every time people stream movies, socialize on social media, shop online, or make online payments amongst others, data is generated. Institutions collect this type of data to be able to analyze and understands customer behavior to be able to improve services, make better decisions and come up with future predictions depending on the trend.

The collected data is raw and has little value unless it is processed and analyzed. This brings about Data Analytics which involves collecting, cleaning, transforming and interpreting data to uncover useful insights.

To be able to perform data analytics processes, the analysts rely on programming tools and one of the most used programming language in data analytics is Python. It has become a favorite among beginners and professionals because it is simple to learn, powerful and supported by rich ecosystem of libraries designed for data analysis.

This article will cover what is python, why it is widely used in data analytics, the key libraries every beginner should learn, how it helps in cleaning and analyzing data and why python is the best choice for professions in data analytics.

What Is Python
Python is a programming language created by Guido Van Rossum and was first released in 1991.

Unlike some programming languages that require complex syntax, python uses clean and straightforward commands that resemble plain English.

Example of python command

print("Hello, World!")

The simple command line displays text on the screen.

Python is known for:

Large Community support
Versatility
Simplicity

Why Python Is Popular in Data Analytics
Python has become one of mostly used tools in data analytics for several reasons.

Easy to Learn and Use
Data analysis involves solving business and technical problems. Analyst should focus on understanding data rather than struggling with difficult programming syntax.

Python's simple structure allows beginners to write meaningful programs easily.

Example:
Calculating average using python

nums = [10, 20, 30, 40]
avg = sum(nums) / len(nums)
print(avg)

The simple structure of python makes it ideal for people transitioning into analytics.

Libraries Ecosystem
Python provides specialized libraries that simplify data-related task.

Strong Data Handling Capabilities

Python can process:

Unstructured data (text, images)
Semi_structured data (JSON,XML)
Structured data (tables, spreadsheets)

This flexibility makes it useful across many industries.

Integration with Other Tools

Python works well with:

Jupyter Notebook
MS Excel
MS Power BI
MySQL
PostgreSQL

This allows analyst to build complete workflows

High Industry Demand
Many companies actively seek python skilled analysts because it helps automate repetitive tasks and process large dataset efficiently.

Industries using python includes:

Finance
E-commerce
Healthcare
Marketing
Education
Telecommunications

Python Libraries Used in Data Analytics.

One of python's greatest strength is its libraries

A library is a collection of pre-written code that performs specific tasks. Some of most important libraries for beginners include:

Pandas
Pandas is the most widely used library for data manipulation and analysis.
It helps analysts:

Read dataset
Clean data
filter rows
Handle missing values
Group and summarize data

Example:

import pandas as pd

data = pd.read_csv("sales.csv")
print(data.head())

This loads a CSV file and displays the first five rows.
Pandas is essential for any data analyst.

NumPy
NumPy is used for numerical operations.
It is useful for:

Mathematical calculations
Working with arrays
Statistical analysis

Example:

import numpy as np

nums = np.array([10, 20, 30])
print(np.mean(nums))

Matplotlib
This library is used for creating graphs and charts.

Example:

import matplotlib.pyplot as plt

plt.plot([1,2,3],[4,5,6])
plt.show()

It helps analysts visualize trends

Seaborn
Seaborn build on Matplotlib and creates more attractive visualizations
It is commonly used for:

Heatmaps
Bar charts
Distribution

Scikit_learn
Although mainly used in machine learning, beginners can use it for predictive analytics.
It support:

Regression
Classification
Clustering

Jupyter Notebook
Jupyter notebook allows analysts to write code, visualize results and document analysis in one place.
It is widely used for learning and experimentation.

How Python Is Used to Clean, Analyze and Visualize Data.

Data Cleaning
Raw data is usually messy, common problems include:

Missing values
Duplicates records
Incorrect formats
Typographical errors

Python helps to clean such problems in data efficeintly.

Example:

import pandas as pd

data = pd.read_csv("customers.csv")

data.drop_duplicates(inplace=True)
data.fillna(0, inplace=True)

This script removes duplicates and fills missing values.
Cleaning data is important because poor-quality data leads to inaccurate analysis.

Data Analysis
After cleaning the dataset, analysts explore the data to identify patterns

Python can calculate:

Averages
Totals
Trends
Correlations

Example:

sales.groupby("Region")["Revenue"].sum()

This script calculates total revenue by region.
Analysts use such insights to answer business questions.

For Example:

Which product sells the most
Which customer segment is most profitable
Which month generates highest or lowest revenue

Data Visualization
Visualizations makes insights easier to understanda.
Instead of reading large tables, decision-makers can quickly interpret charts.

Example:

import seaborn as sns

sns.barplot(x="Region", y="Revenue", data=sales)

This creates a bar chart showing regional revenue.

Python supports:

Line charts
Pie charts
Scatter plots
Histograms
Heatmaps Visualization is critical because it helps communicate findings clearly

Real-World Examples of Python in Data Analytics

Python is widely used in real-world organizations.

E-Commerce
Online stores analyze customer purchase behaviour

Python helps answer:

Which product sells most
Which products are often bought together
Which customer are likely to return Companies like Alibaba use data analytics extensively.

Finance
Banks and financial institutions use python for:

Customer segmentation
Risk analysis
Fraud detection By analyzing transaction patterns, suspicious activity can be detected quickly.

Healthcare
Hospitals use python to analyze:

Patient records
Disease trends
Treatment outcomes This improves decision-making and patient care

Marketing
Business analyst analyze business performance using python.

Questions include:

Which audience engages most?
Which advertisements perform best?
What is the conversion rate?

Sports Analytics
Sports teams analyze players or club performance and match statistics. Python helps identify strengths and weaknesses. This helps improve team strategies.

Why Beginners Should Learn Python.

If you are new to data analytics, python is one of the best starting points.

Beginner-Friendly
Its syntax is simple and readable.
You can start solving real problems quickly.

Strong Career Opportunities
Python is highly valued in roles such as:

Data Analyst
Data Scientist
Business Analyst
Machine Learning Engineer Learning python increases employability.

Supports Career Growth
Once you master Python for analytics, you can expand into:

Machine Learning
Artificial intelligence
Data Engineering
Automation Python opens many career paths.

Practical and In-Demand
Python is not just theoretical.
You can immediately apply it to real datasets and projects.
This makes learning more engaging and rewarding.

Conclusion

In modern data analytics, python has become one of most important tools.

With python, analyst can:

Clean messy datasets
Analyze trends and patterns
Create meaningful visualizations
Generate actionable business insights

Python powers real world data driven decisions across industries such as E-commerce, Finance, Healthcare and sports.

Learning python as a beginner in data analytics profession provides a strong technical foundation and opens doors to exciting career opportunities in the growing field of data.

As data continues to shape the future, python remains one of the tools to help analysts transform raw information into valuable knowledge.

Understanding SQL Joins and SQL Functions, CTEs and Subqueries.

Joseous Ng'ash — Tue, 05 May 2026 23:29:35 +0000

As my journey in becoming a competent data analytics, my SQL knowledge continues to deepen and as a result, I also publish few things pick up through the process.
couple of weeks back I published about SQL fundamentals, covering that is DDL, DML and Data Manipulation. Read more about SQL fundamentals from this link Click here to visit dev.to.

Building on SQL skills and data analysis, I have come to know you can work on different tables at the same time through the help of SQL joins and SQL Functions.

What is Join?

A JOIN in SQL is used to link or combine rows from two or more tables based on related column between them and it is usually a Primary Key and Foreign Key.
Think of it like, one table has students and another has scores, JOIN will help you see which student took which course.

Types of Joins

LEFT JOIN (LEFT OUTER JOIN) This type of returns all records from Left table and matching records from the right table

Example:

SELECT s.name, c.course_name
FROM students s
LEFT JOIN courses c
ON s.course_id = c.id;

All students appear, even if they are not assigned a course.

INNER JOIN
This type of JOIN returns only matching records from both tables.

Example:

SELECT s.name, c.course_name
FROM students s
INNER JOIN courses c
ON s.course_id = c.id;

RIGHT JOIN (RIGHT OUTER JOIN)
The JOIN returns all records from the right table and matching ones from the left.

Example

SELECT s.name, c.course_name
FROM students s
RIGHT JOIN courses c
ON s.course_id = c.id;

All courses appear, even if no student is enrolled.

FULL JOIN(FULL OUTER JOIN)
This type of JOIN returns all records when there is a match in either table.

Example:

SELECT s.name, c.course_name
FROM students s
FULL JOIN courses c
ON s.course_id = c.id;

JOINs are about relationships between tables
Without JOINs, databases would be much less powerful

SQL Window Functions

What are Window Functions?
Window functions are used to calculate across rows without collapsing them.

Example:

SELECT name, department, salary,
       AVG(salary) OVER (PARTITION BY department) AS avg_salary
FROM employees;

The output from this query, every employee still appears, which also includes department average.

SQL functions Every Beginner Should Know

COUNT():

The function counts rows present in a given table.

Example:

SELECT COUNT(*) FROM students;

Counts total number of students

SUM():

This function is used to add numeric values.

Example:

SELECT SUM(salary) FROM employees;

This will get the total salary.

AVERAGE():

The function is used to get Average values like school exam result performance.

Example:

SELECT AVG(marks) FROM exams;

The output will give the average marks.

UPPER()/LOWER():

The function is used to get or change text case,
UPPER() is used to change text into upper case while
LOWER() is used to change text into lower case.

Example:

SELECT 
   UPPER(first_name),
   LOWER(last_name) 
FROM students;

The student's first name will be in upper case and second name will be in lower case.

NOW()/CURRENT_DATE():
This function is used to get current date or time. It is useful when filtering recent records.

Example:

SELECT CURRENT_DATE;

In SQL functions are many, the highlighted functions are most common and every beginner should know and understand how they work in order to ease the work as data analysts, data engineer or scientist.

SQL Subqueries

What is Subquery in SQL:
A Subquery is a query written inside another SQL Query, It executes first and its result is used by the outer query.

Example:

SELECT name, salary
FROM employees
WHERE salary > (
    SELECT AVG(salary)
    FROM employees
);

The inner query calculates average salary and the outer query compares the employee's salary against average

Types of Subquery
We have different types of subqueries, they include;

Scalar Subquery: It is used to return single value.

Example:

SELECT *
FROM products
WHERE price > (
    SELECT AVG(price)
    FROM products
);

Multiple-row Subquery: This subquery returns multiple rows as its name suggests.

Example:

SELECT name
FROM employees
WHERE department_id IN (
    SELECT id
    FROM departments
);

Correlated Subquery: This query references one or more columns from the outer(main) query, it depends on the outer query.

Example:

SELECT e1.name, e1.salary
FROM employees e1
WHERE salary > (
    SELECT AVG(e2.salary)
    FROM employees e2
    WHERE e1.department = e2.department
);

The query compares each employee to the average salary in their department.

Common Table Expression (CTE)

What is CTE?
A CTE (Common Table expression) is a temporary named result set created using the With clause.
CTE makes query easier to organize and read.

Example: same query using a CTE

WITH avg_salary AS (
    SELECT AVG(salary) AS avg_sal
    FROM employees
)

SELECT name, salary
FROM employees
WHERE salary > (
    SELECT avg_sal
    FROM avg_salary
);

Reasons for using CTEs

The need for better readability
Simply complex queries
The need for organized structure
If you need to reuse intermediate results

CTE with Multiple Steps
One advantage of CTEs is chaining logic

Example: Monthly sales analysis

WITH monthly_sales AS (
    SELECT
        DATE_TRUNC('month', sale_date) AS month,
        SUM(amount) AS revenue
    FROM sales
    GROUP BY month
),

ranked_sales AS (
    SELECT *,
           RANK() OVER (ORDER BY revenue DESC) AS sales_rank
    FROM monthly_sales
)

SELECT *
FROM ranked_sales;

This will calculate monthly revenue, rank months by revenue and return final results.

SQL Learning Roadmap for a Beginner

SQL Fundamentals

JOINs
Group BY
Window Functions
SQL Functions
Subqueries
CTEs

These will be core concepts needed for:

Preparing for SQL technical interview
Writing quality SQL scripts
Dashboard preparation for tools like Ms Power BI
Data analysis

Conclusion

Since now as a data scientist, data manipulation and analysis is easier now that I have learnt and understood SQL Functions, Window Function and also Joins.

Also the use of CTEs and Subquery makes SQL query to be easier to read and organization.
Note: If a subquery starts to become difficult to read, convert it into a CTE

Understanding SQL Fundamentals: DDL, DML, and Data Manipulation.

Joseous Ng'ash — Mon, 13 Apr 2026 11:27:48 +0000

What is SQL(Structured Query Language):

Structured Query Language (SQL) is a powerful tool used to manage and manipulate data in relational databases.

SQL commands are categorized into different groups, two of the most important being DDL and DML.

What is DDL (Data Definition Language)

This refers to commands used to define and manage the structure of a database. These commands include creating, modifying, or deleting database objects such as tables.

Examples of DDL Commands:

CREATE – used to create new tables and databases
ALTER – used to modify existing database structures
DROP – used to delete tables or databases

What is DML (Data Manipulation Language)

This refers to commands that deal with managing data within those structures. It allows users to insert, update, retrieve, and delete records.
Examples of DML Commands:

INSERT – adds new records to a table
UPDATE – modifies existing data
DELETE – removes records from a table
SELECT – retrieves data

How I Used CREATE, INSERT, UPDATE, and DELETE

In the assignment, I applied several SQL commands to interact with the database.

CREATE

I used the CREATE statement to define tables such as students, subjects, and exam_results. This involved specifying column names, data types, and constraints like primary keys.

CREATE TABLE exam_results (
    result_id INT PRIMARY KEY,
    student_id INT,
    subject_id INT,
    marks INT,
    exam_date DATE,
    grade VARCHAR(2)
);

INSERT

I used the INSERT command to add records into the tables. For example, I inserted student details such as names, city, class, gender, and date of birth into the students table.

insert into students(student_id, first_name, last_name, gender,
date_of_birth, class, city)
values 
(1, 'Amina', 'Wanjiku', 'F', '2008-03-12', 'Form 3', 'Nairobi'),
(2, 'Brian', 'Ochieng', 'M', '2007-07-25', 'Form 4', 'Mombasa'),
(3, 'Cynthia', 'Mutua', 'F', '2008-11-05', 'Form 3', 'Kisumu'),
(4, 'David', 'Kamau', 'M', '2007-02-18', 'Form 4', 'Nairobi'),
(5, 'Esther', 'Akinyi', 'F', '2009-06-30', 'Form 2', 'Nakuru'),
(6, 'Felix', 'Otieno', 'M', '2009-09-14', 'Form 2', 'Eldoret'),
(7, 'Grace', 'Mwangi', 'F', '2008-01-22', 'Form 3', 'Nairobi'),
(8, 'Hassan', 'Abdi', 'M', '2007-04-09', 'Form 4', 'Mombasa'),
(9, 'Ivy', 'Chebet', 'F', '2009-12-01', 'Form 2', 'Nakuru'),
(10, 'James', 'Kariuki', 'M', '2008-08-17', 'Form 3', 'Nairobi');

UPDATE

I used UPDATE to modify existing records. For example, updating student_id = 5 city from Nakuru to Nairobi.

UPDATE students
SET city = 'Nairobi'
WHERE student_id = 5;

DELETE

The DELETE statement was used to remove result_id = 9, which was cancelled, ensuring data accuracy and consistency.

DELETE FROM exam_results
WHERE result_id = 9;

These commands helped simulate real-world database operations, where data is constantly being created, updated, and maintained.

Using WHERE to Filter Data

The WHERE clause is used to filter records based on specific conditions. It ensures that queries return only relevant data instead of the entire dataset.

Some Commonly Used Operators Include:

= (equals): used to match exact values Example:

DELETE FROM exam_results
WHERE result_id = 9;

sql

> (greater than): used for numeric comparisons Example:

SELECT *
FROM exam_results
WHERE marks > 70;

BETWEEN: used to filter values within a range Example:

SELECT *
FROM exam_results
WHERE exam_date BETWEEN '2024-03-15' AND '2024-03-18';

IN: used to match multiple values Example:

SELECT *
FROM students
WHERE city IN ('Nairobi', 'Mombasa', 'Kisumu');

LIKE: used for pattern matching Example:

SELECT *
FROM students
WHERE first_name LIKE 'A%' OR first_name LIKE 'E%';

Using the WHERE clause makes queries more efficient and meaningful by narrowing down results.

Using CASE WHEN to Transform Data

The CASE WHEN statement is used to apply conditional logic within SQL queries. It allows transformation of data into more meaningful categories.

For example, I used CASE WHEN to categorize students' grades:

Question:

Write a query using CASE WHEN to label each exam result with a grade description:

'Distinction' if marks >= 80
'Merit' if marks >= 60
'Pass' if marks >= 40
'Fail' if marks below 40

SELECT *,
CASE
    WHEN marks >= 80 THEN 'Distinction'
    WHEN marks >= 60 THEN 'Merit'
    WHEN marks >= 40 THEN 'Pass'
    ELSE 'Fail'
END AS performance
FROM exam_results;

This approach helps in creating readable insights from raw data, making it easier to analyze and present findings.

Reflection

Learning SQL has been both challenging and rewarding.

One of the main challenges was understanding how different SQL commands interact, especially when updating or filtering data. Small syntax errors, such as missing quotes or incorrect data types, often led to errors.

However, I found it very interesting how SQL allows you to manipulate and transform data efficiently. The CASE WHEN clause, in particular, stood out as a powerful tool for turning raw data into meaningful insights. Additionally, learning how to filter data using different operators improved my ability to write more precise and clear queries.

In the end, this experience strengthened my understanding of database operations and improved my confidence in using SQL for data analysis.

A comprehensive guide to Publishing And Embedding Power BI Reports on Website with iFrames.

Joseous Ng'ash — Mon, 06 Apr 2026 17:11:50 +0000

What exactly is Microsoft Power BI?

In simple and understandable terms, it refers to business analytic tool by Microsoft that helps you visualize data, create reports and share insights.

Key components

Power BI Desktop – Build reports on your computer.
Power BI Mobile – View reports on phones/tablets.
Power BI Service – Cloud platform for sharing and collaboration.

Why Use Power BI?

It provides real time analytics
It is easy to share and embed
It can connect to multiple data sources
Drag and Drop dashboards

Power BI Publishing

Power BI publishing means moving your report from Power BI Desktop to Power BI Service(cloud) so others on the same team can access it.

Power BI Report Publishing Overview

Highlight Of Steps Involved In Power BI Report Publishing

Create report in Power BI Desktop
Save your report
Click publish on Power BI desktop
Sign in to publish
Choose workspace
Upload to Power BI Service
Access via browser

Detailed Power BI Publishing steps

STEP ONE

Create Report In Power BI Desktop
To create Power BI reports, you have to import data, clean and transform the data then build visuals.
Importing Data; Data can be imported from different sources, eg excel as flat file or excel workbook, different database, SQL server.
One way to import data, open Power BI desktop and it will open to home page where you have different options to select from to get data.
Home Panel with different data sources

Another way to import data to create Power BI reports, click on Get Data on the far left on navigation panel/home panel and click on your preferred data source or depending on where the data you are working with is saved.

Get Data Option to import data to power BI desktop

After selecting your preferred data source, click connect located on the bottom right of Get Data navigation panel and wait for the data to be connected from source then click the connected table to select and preview the data.
In this demo I used an Excel Workbook as my data source.

What to expect after connecting to your preferred data source, my table is called Sheet1 containing detailed imported data

Data Cleaning And Transformation(Power Query)
Raw data is often messy -- Missing Values, Inconsistent format, Duplicates. This process of data cleaning and transforming ensures the report is accurate and reliable. In simple terms data cleaning and transforming in Microsoft power BI is the process of preparing raw data so it becomes accurate, consistent, and ready for analysis.

Data Cleaning Involves

Removing Duplicates: This ensures each record is unique and avoid double counting during analysis. To start data cleaning, on the navigator on the bottom right click on Transform option which will take you direct to Power Query What is Power Query: It is a tool used to connect, clean, transform, and load data into Power BI for analysis.

Transform Option On Navigator

Power Query Interface

Main sections in Power Query
1️⃣ Ribbon(Top Menu): It gives access to all transformation tools and commands, key tabs includes;

Home → load data, close and apply
Transform → Change data types, pivot/unpivot
View → Toggle panels (like formula bar)
Add Column → Create calculated columns

2️⃣ Queries Pane (Left Side): it contains a list of all tables which help in navigating between different datasets and organize queries(rename, group or duplicates)

3️⃣ Data Preview (Center Table View): it serve the purpose of viewing and interacting with your dataset and applying transformation directly on columns.

4️⃣ Applied Steps Pane (Right Side): it shows all applied steps during data cleaning and transformation like removed columns, filter row or change type.

5️⃣ Query Settings Panel (Right Side – Top): it is located on top of applied steps, which is used to rename query and add description.

6️⃣ Formula Bar: it is used to display formulas and uses M Language

7️⃣ Column Headers: this is the top row of every column which offers quick transformation through dropdown menus.

Handling Missing Data: This helps in replacing missing values with Nulls, Not Provided, Missing, N/A or remove incomplete rows.
Fixing Data Types: Convert columns into correct formats eg number to text or text to date.
Select on the column you want to change data type and right click then select Change type and select to the type you want to change your data into(decimal, data, text, number amongst others) as part of data cleaning.

Renaming Columns: make a column make sense or meanifull.

Data Transformation
This involves reshaping data for analysis and visualization. The process involves different tasks which includes:

Data filtering
Creating new columns
Grouping and aggregating
Pivoting and unpivoting
Splitting and merging columns
Data sorting

After the data is imported, cleaned and transformed. The data is ready to be used to Build Visual.
Reports visuals are prepared in Power BI. To make sure the cleaned data is loaded to Power BI, in the Power Query to save changes, click on Home Tab and then click Close and Apply button to save data and load to Power BI directly

Close and Apply Button In Power Query Editor

After Power Query is closed and changes saved, the data is loaded to Power BI to build visual reports.

Report View In Power BI Home Page

It displays table contain our data, panel where visuals appear, filter panel, visualization panel with different charts, a ribbon for different commands and functionality.

Building Visuals In Power BI
After closing Power Query and the cleaned and transformed data is loaded to Power BI, the next step is to use DAX (Data Analysis Expressions) to create necessary measure to help in data analysis.

What Is DAX (Data Analysis Expressions): is a formula language used in Power BI and Analysis Services to perform data calculations and analysis.
Example Of DAX

Profit margin = DIVIDE(SUM(Electronics_sales[Profit]), SUM(Electronics_sales[SalesAmount])) * 100

After we have done the necessary calculations, the next step is choosing the right charts to represent the analysis visually.
Some of the available charts in Power BI include;

Bar Charts
Column charts
Pie charts
Line charts

Charts panel in Power BI

After creating required KPIs and visualizing the report, the next step is publishing to Power BI service.

Complete Power BI Report Looks like demonstrated below.

STEP TWO

Saving Power BI Report(.pbix format)
After you have completed the Power BI Report, the next step is saving the report in the appropriate format.
Always save your Power BI report with extension **.pbix format.
Power BI file save in local machine

STEP THREE

Click Publish
In Power BI desktop home tab Click Publish on the top right of the application.

STEP FOUR

Sign In
After clicking publish button, you will be prompted to Sign In. Use your Microsoft Account to sign in or school email for students and organization account for the organization you are working for for collaborations.
Sign In Window from Power BI Desktop

STEP FIVE

Choose Workspace
After you have provided the Sign In credentials, next you choose the workspace already created from Power BI Service.
Window to choose your workspace from

STEP SIX

Upload to Power BI Service
After choosing workspace, click select to upload to Power BI Srvice.
Select Option

🌐 Embedding Power BI Reports on the Web (iFrame)

After publishing the Power BI report from Desktop, this are the step involved in Embedding report on the web

Go to Power BI service and sign in with your account information.
Go to Workspace where you published power bi file and open your report. Published Power BI report on Power BI service

Click: File → Embed report → Website or Portal

Confirm and generate embed code

Copy the generated iFrame code to code editor like Vs code Copied code in VS code

Save the embedding code to a folder in your PC and open the index.html using your preferred browser and share the link to your team or organization to access the Power BI report.

Conclusion

Creating visuals in Power BI desktop is just the beginning which equips you with data understanding skills and data analysis to derive business insights. After the report is complete, it is only accessible to you and any other person using your computer because it is only available into local machine. To make sure all involved parties get to see the final report for decision making, you have to go an extra mile into publishing it to Power BI service and share the link to the web to relevant parties. For any individual who aspire to be data analyst, it is important to understand the process of publishing the report and embedding into a website.

Power BI Data Modeling.

Joseous Ng'ash — Tue, 31 Mar 2026 08:02:15 +0000

What is Data Modeling in Power BI

In simple and understandable terms, Data Modeling is simply a way to organize, connect and structure your data in a way it can be easily analyzed and visualized.

Raw data = Messy datasets
Data modeling = organize and arrange the dataset into meaningful structure

Data Modeling in Power BI: Creating relationships between tables and structuring data into a format that enables accurate analysis and reporting.

Data Modeling Involves

Creating relationship
Defining structure(Schema)
Organizing Data into tables
Setting rules(Cardinality & Filters)

Data modeling is important because;

We can derive accurate analysis
We can get better insights
Can create clear and accurate dashboards

Power BI uses some of components to help in data modeling

Main Components of Data Modeling in Power BI.

Joins In Power BI
Joins are used when combining tables into one (This is usually done in Power Query Editor).

Types Of Joins In Power BI

Inner Join: It only returns matching rows in both tables and it is used when you only want related data. How Inner join looks like.

Left Outer Join: This is the most commonly used join. It returns all rows from the left table plus matching rows from the right.

Right Outer Join: This join act opposite of the Left Outer Join. It returns all rows from the right table plus matching rows from the left table.

Full Outer Join: This type of join in power bi, returns all the rows from both table, matched or unmatched.

Anti Join(Left/Right): This type of joins are used to return the unmatched rows, usually help in finding missing data.

Relationships In Power BI
In Relationship tables are connected without merging them. Instead of combining data physically, Power BI links tables logically.

Key Concept

Primary Key → Unique column (e.g., ProductID in Products)
Foreign Key → Repeated column (e.g., ProductID in Sales)

Cardinality: defines how table relate, example;

One-to-many(1:):* commonly used relationship, meaning one product many sales.
Many-to-one(:1):* it is the same as one-to-many but reversed.
One-to-one(1:1): this relationship is rarely used during data modeling.
Many-to-many(:): this type of relationship create ambiguity and should be used carefully.

Cross Filter Direction

Single direction: the filter flow from one direction and its the recommended filter.
Both Direction: the filter flows both ways though it can cause confusion if misused.

Schemas (Data Modeling Structures)
Schemas define how your tables are organized.
Star Schema (best practice)
This is the recommended approach in Power BI.

Structure
Fact Table: contains measurable data eg sales, revenue, quantity.
Dimension Tables Eg customers, dates, products

Benefits of star schema includes;

It is easy to understand
Better performance in Power BI
It is easy and fast

Snowflake Schema
This is a more complex version of star schema

Structure
In this type of schema, dimension tables are split into Sub-Tables though it is slow in performance and harder to manage.

Flat Table (Single Table)
This type, the data is in one table and has poor performance, hard to manage and also data redundancy.

Conclusion

Data modeling in Power BI is the process of organizing data into structured tables and creating relationships between them to enable accurate analysis and reporting. It involves arranging data in an efficient format, such as a star schema, to improve performance and ensure that filters and calculations work correctly. Overall, good data modeling helps transform raw data into meaningful insights for better decision-making.

Excel Real world Applications

Joseous Ng'ash — Fri, 27 Mar 2026 11:44:09 +0000

HOW EXCEL IS USED IN REAL WORLD DATA ANALYSIS

What really is Excel, in simple terms it is just an application from Microsoft to help ordinary people to keep their records either groceries records, building materials records, names of people traveling, basically anything involving records keeping for future references.
But for professionals like Data Analytics it is a software used to organize, analyze and visualize data. It is like a digital notebook comprising of columns and rows where you can:

create charts
perform calculations
store data of different data type eg; text, dates, numbers

Excel worksheet is made of collection of cells.

What you can do with Excel

Data analysis
Data Visualization
Automation
Data Organization
Perform Calculations

How Excel is Used in Real-world:

Track monthly expenses either by an individual or in an industrial set up.
Widely used in finance, engineering, business and data analysis.
Used to clean and prepare data before its used in other tools like SQL and Python.
It is an essential skill for careers like accountant, data engineer and data analyst.
Analyzing products performance and help in decision making in a company.
Create reports to derive clear insights.

Key Features found in Microsoft Excel

Charts And Graphs

Charts and graphs are used for data visualization which help in creating dashboard reports, some examples of charts and graphs includes:

Bar Charts
Line Graphs
Pie Charts

Example of a Pie and Bar Chart

An example of dashboard reports created from different charts and graphs

Sorting And Filtering

It is a feature used to Analyze large datasets quickly, you can either sort data from largest to smallest, smallest to largest or filter specific data.
Filter marks are always located on top of every column but are activated from the tab editing under sort and filter.

Filter Marks on every column to filter specific data

Pivot Tables

It is one of the powerful feature used for analysis in Excel

Example of how pivot table is used in data analysis

Conditional Formatting

The feature automatically changes the color of cell based on highlight rule either;

High/Low values
Duplicates
Trends Example of conditional formatting used to highlight top most performing product, using Top/Bottom Rule

Data Validation

The feature helps to keep data clean and consistent.

Data Organization

Helps make the data readable and manageable by;

Storing data in rows and columns
Format as table
structured referencing

Formulas And Functions

This feature helps excel to perform calculations using formulas
Formulas and functions are excel built-in features which are activated by typing equal sign (=) followed by specific formula or function example (VLOOKUP()) on any excel cell containing data to be filtered, aggregated or edited.

Basic operators in Excel: +,-,*,/
Functions:
- SUM() adds Values
- AVERAGE() finds mean
- IF() logical decisions
- VLOOKUP()/XLOOKUP() search data

Import And Export Data

With this feature will help in importing data from CSV files and Databases, and Export to a different formats

Conclusion
Learning Microsoft excel has transformed how I understand and interpret data, before data felt overwhelming, I would look at dataset without knowing how to extract useful insights.

Features like formulas and functions helped move far from manual calculations and start focusing on what dataset actually represent.

features like filtering, pivot table and sorting showed me data is not just static, it can be explored from different perspectives to reveal hidden insights.

Excel has also made me improve on data handling, the importance of missing values, ensuring data validation. Working with any dataset has made me more careful knowing small errors can lead to incorrect conclusions.

Overall, learning Excel has helped in a way that when I look at the data , I can understand, question and develop analytical reports to the organization for better decision making.